2026.05.06 08:48:25.244226 [ 31631 ] {} <Information> Application: Will watch for the process with pid 31633
2026.05.06 08:48:25.247391 [ 31633 ] {} <Information> Application: Forked a child process to watch
2026.05.06 08:48:25.249375 [ 31633 ] {} <Information> StatusFile: Status file /var/run/clickhouse-server/clickhouse-server.pid already exists - unclean restart. Contents:
2722
2026.05.06 08:48:25.250781 [ 31633 ] {} <Information> SentryWriter: Sending crash reports is disabled
2026.05.06 08:48:26.163920 [ 31633 ] {} <Information> Application: Starting ClickHouse 24.8.14.10546.altinitytest (revision: 4294967295, git hash: bd28307cd2efb6638c150abd8a7331dbf458804a, build id: DA078E6F75C2DC04F72E549EE0FB8F2D93941D6A), PID 31633
2026.05.06 08:48:26.165623 [ 31633 ] {} <Information> Application: starting up
2026.05.06 08:48:26.165842 [ 31633 ] {} <Information> Application: OS name: Linux, version: 5.15.0-130-generic, architecture: x86_64
2026.05.06 08:48:26.214794 [ 31633 ] {} <Warning> Context: Server was built with memory sanitizer. It will work slowly.
2026.05.06 08:48:26.215123 [ 31633 ] {} <Information> Application: Available RAM: 30.59 GiB; logical cores: 16; used cores: 16.
2026.05.06 08:48:26.215218 [ 31633 ] {} <Information> Application: Available CPU instruction sets: SSE, SSE2, SSE3, SSSE3, SSE41, SSE42, F16C, POPCNT, BMI1, BMI2, PCLMUL, AES, AVX, FMA, AVX2, AVX512F, AVX512DQ, AVX512IFMA, AVX512CD, AVX512BW, AVX512VL, AVX512VBMI, AVX512VBMI2, SHA, ADX, RDRAND, RDSEED, RDTSCP, CLFLUSHOPT, CLWB, XSAVE, OSXSAVE
2026.05.06 08:48:26.215284 [ 31633 ] {} <Information> Application: Query Profiler is disabled because it cannot work under sanitizers when two different stack unwinding methods will interfere with each other.
2026.05.06 08:48:26.215394 [ 31633 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2026.05.06 08:48:26.217616 [ 31633 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/thermal
2026.05.06 08:48:26.217690 [ 31633 ] {} <Trace> AsynchronousMetrics: Scanning /sys/block
2026.05.06 08:48:26.217914 [ 31633 ] {} <Trace> AsynchronousMetrics: Scanning /sys/devices/system/edac
2026.05.06 08:48:26.217981 [ 31633 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/hwmon
2026.05.06 08:48:26.218217 [ 31633 ] {} <Information> StatusFile: Status file /var/lib/clickhouse/status already exists - unclean restart. Contents:
PID: 2722
Started at: 2026-05-06 07:50:53
Revision: 4294967295

2026.05.06 08:48:26.235631 [ 31633 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2026.05.06 08:48:26.242076 [ 31633 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2026.05.06 08:48:26.243430 [ 31633 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2026.05.06 08:48:30.732083 [ 31633 ] {} <Information> Application: Integrity check of the executable successfully passed (checksum: 5FAC079CF9229F28615B0B9572615813)
2026.05.06 08:48:30.733464 [ 31633 ] {} <Trace> Application: Will do mlock to prevent executable memory from being paged out. It may take a few seconds.
2026.05.06 08:48:30.733567 [ 31633 ] {} <Trace> Application: The memory map of clickhouse executable has been mlock'ed, total 1019.53 MiB
2026.05.06 08:48:30.733822 [ 31633 ] {} <Debug> Application: Set max number of file descriptors to 1048576 (was 1024).
2026.05.06 08:48:30.733874 [ 31633 ] {} <Debug> Application: rlimit on number of threads is 18446744073709551615
2026.05.06 08:48:30.733937 [ 31633 ] {} <Debug> Application: Initializing DateLUT.
2026.05.06 08:48:30.733985 [ 31633 ] {} <Trace> Application: Initialized DateLUT with time zone 'Europe/Amsterdam'.
2026.05.06 08:48:30.734132 [ 31633 ] {} <Debug> Context: Setting up /var/lib/clickhouse/tmp/ to store temporary data in it
2026.05.06 08:48:30.734493 [ 31633 ] {} <Trace> TLDListsHolder: public_suffix_list loading from /etc/clickhouse-server/top_level_domains/public_suffix_list.dat
2026.05.06 08:48:30.749731 [ 31633 ] {} <Information> TLDListsHolder: public_suffix_list was added (9084 hosts)
2026.05.06 08:48:30.749845 [ 31633 ] {} <Trace> TLDListsHolder: no_new_line_list loading from /etc/clickhouse-server/top_level_domains/no_new_line_list.dat
2026.05.06 08:48:30.751043 [ 31633 ] {} <Information> TLDListsHolder: no_new_line_list was added (1 hosts)
2026.05.06 08:48:30.751506 [ 31633 ] {} <Debug> Application: Configuration parameter 'interserver_http_host' doesn't exist or exists and empty. Will use 'e033c3a7a850' as replica host.
2026.05.06 08:48:30.751585 [ 31633 ] {} <Debug> Application: Initializing interserver credentials.
2026.05.06 08:48:30.752052 [ 31633 ] {} <Trace> NamedCollectionsMetadataStorage: Using local storage for named collections at path: /var/lib/clickhouse/named_collections
2026.05.06 08:48:30.755337 [ 31633 ] {} <Test> NamedCollectionFactory: Loaded 9 collections from config
2026.05.06 08:48:30.755406 [ 31633 ] {} <Test> NamedCollectionFactory: Loaded 0 collections from sql
2026.05.06 08:48:30.755835 [ 31633 ] {} <Information> SensitiveDataMaskerConfigRead: 1 query masking rules loaded.
2026.05.06 08:48:30.758048 [ 31633 ] {} <Information> CgroupsMemoryUsageObserver: Will read the current memory usage from '/sys/fs/cgroup/' (cgroups version: v2), wait time is 15 sec
2026.05.06 08:48:30.768529 [ 31633 ] {} <Debug> ConfigReloader: Loading config '/etc/clickhouse-server/config.xml'
2026.05.06 08:48:30.768608 [ 31633 ] {} <Debug> ConfigProcessor: Processing configuration file '/etc/clickhouse-server/config.xml'.
2026.05.06 08:48:30.773303 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/CORS.xml'.
2026.05.06 08:48:30.774428 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/azure_storage_conf.xml'.
2026.05.06 08:48:30.775964 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/backoff_failed_mutation.xml'.
2026.05.06 08:48:30.776415 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/backups.xml'.
2026.05.06 08:48:30.777096 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/blob_storage_log.xml'.
2026.05.06 08:48:30.777664 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/block_number.xml'.
2026.05.06 08:48:30.778147 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/cannot_allocate_thread_injection.xml'.
2026.05.06 08:48:30.778551 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/clusters.xml'.
2026.05.06 08:48:30.785697 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/compressed_marks_and_index.xml'.
2026.05.06 08:48:30.786214 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/core.xml'.
2026.05.06 08:48:30.786719 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/custom_disks_base_path.xml'.
2026.05.06 08:48:30.787147 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/custom_settings_prefixes.xml'.
2026.05.06 08:48:30.787623 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/database_atomic.xml'.
2026.05.06 08:48:30.788161 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/database_catalog_drop_table_concurrency.xml'.
2026.05.06 08:48:30.788769 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/disable_s3_env_credentials.xml'.
2026.05.06 08:48:30.789434 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/display_name.xml'.
2026.05.06 08:48:30.790055 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/enable_access_control_improvements.xml'.
2026.05.06 08:48:30.790776 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/enable_keeper_map.xml'.
2026.05.06 08:48:30.791343 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/enable_wait_for_shutdown_replicated_tables.xml'.
2026.05.06 08:48:30.792079 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/enable_zero_copy_replication.xml'.
2026.05.06 08:48:30.792756 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/encryption.xml'.
2026.05.06 08:48:30.793520 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/filesystem_cache_log.xml'.
2026.05.06 08:48:30.794157 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/filesystem_caches_path.xml'.
2026.05.06 08:48:30.794903 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/forbidden_headers.xml'.
2026.05.06 08:48:30.795628 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/graphite.xml'.
2026.05.06 08:48:30.797043 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/graphite_alternative.xml'.
2026.05.06 08:48:30.799114 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/grpc_protocol.xml'.
2026.05.06 08:48:30.799785 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/handlers.yaml'.
2026.05.06 08:48:30.801548 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/keeper_port.xml'.
2026.05.06 08:48:30.803135 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/legacy_geobase.xml'.
2026.05.06 08:48:30.803713 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/listen.xml'.
2026.05.06 08:48:30.804201 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/logger_trace.xml'.
2026.05.06 08:48:30.804763 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/logging_no_rotate.xml'.
2026.05.06 08:48:30.805303 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/lost_forever_check.xml'.
2026.05.06 08:48:30.805831 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/macros.xml'.
2026.05.06 08:48:30.806513 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/max_concurrent_queries.xml'.
2026.05.06 08:48:30.807016 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/max_num_to_warn.xml'.
2026.05.06 08:48:30.807683 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/max_server_memory_usage.xml'.
2026.05.06 08:48:30.808209 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/merge_tree.xml'.
2026.05.06 08:48:30.808892 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/merge_tree_old_dirs_cleanup.xml'.
2026.05.06 08:48:30.809613 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/merge_tree_settings.xml'.
2026.05.06 08:48:30.810229 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/named_collection.xml'.
2026.05.06 08:48:30.812342 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/nlp.xml'.
2026.05.06 08:48:30.813543 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/prometheus.xml'.
2026.05.06 08:48:30.814308 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/query_masking_rules.xml'.
2026.05.06 08:48:30.815110 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/query_masking_rules.xml_original.xml'.
2026.05.06 08:48:30.815809 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/rocksdb.xml'.
2026.05.06 08:48:30.816447 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/secure_ports.xml'.
2026.05.06 08:48:30.817300 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/serverwide_trace_collector.xml'.
2026.05.06 08:48:30.817886 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/session_log.xml'.
2026.05.06 08:48:30.818517 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/ssl_certs.xml'.
2026.05.06 08:48:30.819252 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/storage_conf.xml'.
2026.05.06 08:48:30.823334 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/storage_conf_02944.xml'.
2026.05.06 08:48:30.824503 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/storage_conf_02961.xml'.
2026.05.06 08:48:30.825983 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/storage_conf_02963.xml'.
2026.05.06 08:48:30.826877 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/system_unfreeze.xml'.
2026.05.06 08:48:30.827502 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/tcp_with_proxy.xml'.
2026.05.06 08:48:30.828123 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/test_cluster_with_incorrect_pw.xml'.
2026.05.06 08:48:30.829273 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/text_log.xml'.
2026.05.06 08:48:30.829951 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/top_level_domains_lists.xml'.
2026.05.06 08:48:30.830568 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/top_level_domains_path.xml'.
2026.05.06 08:48:30.831159 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/transactions.xml'.
2026.05.06 08:48:30.832096 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/validate_tcp_client_information.xml'.
2026.05.06 08:48:30.832694 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/zero_copy_destructive_operations.xml'.
2026.05.06 08:48:30.833424 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/zookeeper_fault_injection.xml'.
2026.05.06 08:48:30.834329 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/zookeeper_log.xml'.
2026.05.06 08:48:30.835028 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/zookeeper_write.xml'.
2026.05.06 08:48:30.934382 [ 31633 ] {} <Debug> ConfigProcessor: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
2026.05.06 08:48:30.938394 [ 31633 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/config.xml', performing update on configuration
2026.05.06 08:48:30.949669 [ 31633 ] {} <Information> Application: Setting max_server_memory_usage was set to 15.30 GiB (30.59 GiB available * 0.50 max_server_memory_usage_to_ram_ratio)
2026.05.06 08:48:30.949772 [ 31633 ] {} <Information> CgroupsMemoryUsageObserver: Set new limits, soft limit: 13.77 GiB, hard limit: 14.53 GiB
2026.05.06 08:48:30.949867 [ 31633 ] {} <Information> Application: Setting merges_mutations_memory_usage_soft_limit was set to 15.30 GiB (30.59 GiB available * 0.50 merges_mutations_memory_usage_to_ram_ratio)
2026.05.06 08:48:30.949930 [ 31633 ] {} <Information> Application: Merges and mutations memory limit is set to 15.30 GiB
2026.05.06 08:48:30.961420 [ 31633 ] {} <Information> BackgroundSchedulePool/BgBufSchPool: Create BackgroundSchedulePool with 16 threads
2026.05.06 08:48:30.967126 [ 31633 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 512 threads
2026.05.06 08:48:31.159474 [ 31633 ] {} <Information> BackgroundSchedulePool/BgMBSchPool: Create BackgroundSchedulePool with 16 threads
2026.05.06 08:48:31.166319 [ 31633 ] {} <Information> BackgroundSchedulePool/BgDistSchPool: Create BackgroundSchedulePool with 16 threads
2026.05.06 08:48:31.186700 [ 31633 ] {} <Test> NamedCollectionFactory: Loaded 9 collections from config
2026.05.06 08:48:31.190376 [ 31633 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/config.xml', performed update on configuration
2026.05.06 08:48:31.190609 [ 31633 ] {} <Trace> ConfigReloader: Config reload interval set to 2000ms
2026.05.06 08:48:31.196031 [ 31633 ] {} <Debug> Context: Keeper server is participant of the main zookeeper cluster, will try to connect to it
2026.05.06 08:48:31.196661 [ 31633 ] {} <Test> ZooKeeper: Availability zones from config: [''], client: ''
2026.05.06 08:48:31.197197 [ 31633 ] {} <Trace> ZooKeeperClient: Adding ZooKeeper host localhost:9181 ([::1]:9181), az: UNKNOWN, priority: 0
2026.05.06 08:48:31.198934 [ 31633 ] {} <Information> Context: Cannot connect to ZooKeeper (or Keeper) before internal Keeper start, will wait for Keeper synchronously
2026.05.06 08:48:31.199052 [ 31633 ] {} <Debug> KeeperDispatcher: Initializing storage dispatcher
2026.05.06 08:48:31.200618 [ 31633 ] {} <Information> KeeperContext: Keeper feature flag FILTERED_LIST: disabled
2026.05.06 08:48:31.200715 [ 31633 ] {} <Information> KeeperContext: Keeper feature flag MULTI_READ: disabled
2026.05.06 08:48:31.200770 [ 31633 ] {} <Information> KeeperContext: Keeper feature flag CHECK_NOT_EXISTS: enabled
2026.05.06 08:48:31.200839 [ 31633 ] {} <Information> KeeperContext: Keeper feature flag CREATE_IF_NOT_EXISTS: enabled
2026.05.06 08:48:31.201146 [ 31633 ] {} <Information> KeeperContext: Disk type 'object_storage' is not supported for Keeper
2026.05.06 08:48:31.201249 [ 31633 ] {} <Information> KeeperContext: Disk type 'cache' is not supported for Keeper
2026.05.06 08:48:31.254942 [ 31633 ] {} <Information> AWSClient: ClientConfiguration: User agent is overridden in the config: ClickHouse 24.8.14.10546.altinitytest (altinity build)
2026.05.06 08:48:31.262458 [ 31633 ] {} <Trace> S3Client: Provider type: Unknown
2026.05.06 08:48:31.262552 [ 31633 ] {} <Trace> S3Client: API mode of the S3 client: AWS
2026.05.06 08:48:31.263946 [ 31633 ] {} <Trace> WriteBufferFromS3: Create WriteBufferFromS3, Details: bucket test, key s3/wqx/irqwwmuwdsuqgnlmbffujefjnzmjb
2026.05.06 08:48:31.265146 [ 31633 ] {} <Trace> WriteBufferFromS3: finalizeImpl WriteBufferFromS3. Details: bucket test, key s3/wqx/irqwwmuwdsuqgnlmbffujefjnzmjb.
2026.05.06 08:48:31.265220 [ 31633 ] {} <Test> WriteBufferFromS3: preFinalize WriteBufferFromS3. Details: bucket test, key s3/wqx/irqwwmuwdsuqgnlmbffujefjnzmjb
2026.05.06 08:48:31.265290 [ 31633 ] {} <Test> WriteBufferFromS3: Making single part upload. Details: bucket test, key s3/wqx/irqwwmuwdsuqgnlmbffujefjnzmjb, size 4
2026.05.06 08:48:31.265941 [ 32233 ] {} <Test> WriteBufferFromS3: writing single part upload started. Details: bucket test, key s3/wqx/irqwwmuwdsuqgnlmbffujefjnzmjb
2026.05.06 08:48:31.275294 [ 32233 ] {} <Trace> WriteBufferFromS3: Single part upload has completed. Details: bucket test, key s3/wqx/irqwwmuwdsuqgnlmbffujefjnzmjb, size 4
2026.05.06 08:48:31.275958 [ 31633 ] {} <Trace> WriteBufferFromS3: Close WriteBufferFromS3. Details: bucket test, key s3/wqx/irqwwmuwdsuqgnlmbffujefjnzmjb.
2026.05.06 08:48:31.280266 [ 31633 ] {} <Debug> S3ObjectStorage: Objects with paths [s3/wqx/irqwwmuwdsuqgnlmbffujefjnzmjb] were removed from S3
2026.05.06 08:48:31.280638 [ 31633 ] {} <Test> DiskObjectStorage(s3_disk): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:31.292252 [ 31633 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pws/olqggyxbqhjgqwqucjuqhwyetdmia (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:31.296516 [ 31633 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pws/olqggyxbqhjgqwqucjuqhwyetdmia (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:31.296619 [ 31633 ] {} <Test> ReadBufferFromS3: Read S3 object. Bucket: test, Key: s3/pws/olqggyxbqhjgqwqucjuqhwyetdmia, Version: Latest, Offset: 2
2026.05.06 08:48:31.302584 [ 31633 ] {} <Debug> S3ObjectStorage: Objects with paths [s3/pws/olqggyxbqhjgqwqucjuqhwyetdmia] were removed from S3
2026.05.06 08:48:31.302695 [ 31633 ] {} <Information> DiskObjectStorage(s3_disk): Starting up disk s3_disk
2026.05.06 08:48:31.302754 [ 31633 ] {} <Information> DiskObjectStorage(s3_disk): Disk s3_disk started up
2026.05.06 08:48:31.304763 [ 31633 ] {} <Information> AWSClient: ClientConfiguration: User agent is overridden in the config: ClickHouse 24.8.14.10546.altinitytest (altinity build)
2026.05.06 08:48:31.311931 [ 31633 ] {} <Trace> S3Client: Provider type: Unknown
2026.05.06 08:48:31.312003 [ 31633 ] {} <Trace> S3Client: API mode of the S3 client: AWS
2026.05.06 08:48:31.326743 [ 31633 ] {} <Debug> S3ObjectStorage: Objects with paths [s3_plain/clickhouse_remove_objects_capability_11a20655-72d2-4e0e-b157-3c8461d3c947] were removed from S3
2026.05.06 08:48:31.327025 [ 31633 ] {} <Test> DiskObjectStorage(s3_plain_disk): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:31.340745 [ 31633 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3_plain/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947 (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:31.346650 [ 31633 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3_plain/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947 (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:31.346730 [ 31633 ] {} <Test> ReadBufferFromS3: Read S3 object. Bucket: test, Key: s3_plain/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947, Version: Latest, Offset: 2
2026.05.06 08:48:31.362282 [ 31633 ] {} <Debug> S3ObjectStorage: Objects with paths [s3_plain/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947] were removed from S3
2026.05.06 08:48:31.362359 [ 31633 ] {} <Information> DiskObjectStorage(s3_plain_disk): Starting up disk s3_plain_disk
2026.05.06 08:48:31.362497 [ 31633 ] {} <Information> DiskObjectStorage(s3_plain_disk): Disk s3_plain_disk started up
2026.05.06 08:48:31.362596 [ 31633 ] {} <Information> KeeperContext: Disk type 'cache' is not supported for Keeper
2026.05.06 08:48:31.362642 [ 31633 ] {} <Information> KeeperContext: Disk type 'cache' is not supported for Keeper
2026.05.06 08:48:31.362677 [ 31633 ] {} <Information> KeeperContext: Disk type 'object_storage' is not supported for Keeper
2026.05.06 08:48:31.362724 [ 31633 ] {} <Information> KeeperContext: Disk type 'object_storage' is not supported for Keeper
2026.05.06 08:48:31.362765 [ 31633 ] {} <Information> KeeperContext: Disk type 'local_blob_storage' is not supported for Keeper
2026.05.06 08:48:31.362798 [ 31633 ] {} <Information> KeeperContext: Disk type 'cache' is not supported for Keeper
2026.05.06 08:48:31.362832 [ 31633 ] {} <Information> KeeperContext: Disk type 'cache' is not supported for Keeper
2026.05.06 08:48:31.362865 [ 31633 ] {} <Information> KeeperContext: Disk type 'cache' is not supported for Keeper
2026.05.06 08:48:31.362898 [ 31633 ] {} <Information> KeeperContext: Disk type 'cache' is not supported for Keeper
2026.05.06 08:48:31.362944 [ 31633 ] {} <Information> KeeperContext: Disk type 'cache' is not supported for Keeper
2026.05.06 08:48:31.364559 [ 31633 ] {} <Information> AWSClient: ClientConfiguration: User agent is overridden in the config: ClickHouse 24.8.14.10546.altinitytest (altinity build)
2026.05.06 08:48:31.371746 [ 31633 ] {} <Trace> S3Client: Provider type: Unknown
2026.05.06 08:48:31.371806 [ 31633 ] {} <Trace> S3Client: API mode of the S3 client: AWS
2026.05.06 08:48:31.385397 [ 31633 ] {} <Debug> S3ObjectStorage: Objects with paths [special/dil/sldycmqtjexwsawptdbfgwfetbjaq] were removed from S3
2026.05.06 08:48:31.385713 [ 31633 ] {} <Test> DiskObjectStorage(s3_no_cache): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:31.397558 [ 31633 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: special/xhn/junoesrwwmmgbfbqvrdceckrobnnc (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:31.401488 [ 31633 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: special/xhn/junoesrwwmmgbfbqvrdceckrobnnc (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:31.401569 [ 31633 ] {} <Test> ReadBufferFromS3: Read S3 object. Bucket: test, Key: special/xhn/junoesrwwmmgbfbqvrdceckrobnnc, Version: Latest, Offset: 2
2026.05.06 08:48:31.407238 [ 31633 ] {} <Debug> S3ObjectStorage: Objects with paths [special/xhn/junoesrwwmmgbfbqvrdceckrobnnc] were removed from S3
2026.05.06 08:48:31.407316 [ 31633 ] {} <Information> DiskObjectStorage(s3_no_cache): Starting up disk s3_no_cache
2026.05.06 08:48:31.407358 [ 31633 ] {} <Information> DiskObjectStorage(s3_no_cache): Disk s3_no_cache started up
2026.05.06 08:48:31.409071 [ 31633 ] {} <Information> AWSClient: ClientConfiguration: User agent is overridden in the config: ClickHouse 24.8.14.10546.altinitytest (altinity build)
2026.05.06 08:48:31.416908 [ 31633 ] {} <Trace> S3Client: Provider type: Unknown
2026.05.06 08:48:31.416966 [ 31633 ] {} <Trace> S3Client: API mode of the S3 client: AWS
2026.05.06 08:48:31.430283 [ 31633 ] {} <Debug> S3ObjectStorage: Objects with paths [test_02944/vdy/byhagshfckrapuhrjyeclgifubwnt] were removed from S3
2026.05.06 08:48:31.430610 [ 31633 ] {} <Test> DiskObjectStorage(s3_disk_02944): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:31.442738 [ 31633 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: test_02944/qwm/xryamieayuacqartbhkkzsuffepfc (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:31.446869 [ 31633 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: test_02944/qwm/xryamieayuacqartbhkkzsuffepfc (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:31.446952 [ 31633 ] {} <Test> ReadBufferFromS3: Read S3 object. Bucket: test, Key: test_02944/qwm/xryamieayuacqartbhkkzsuffepfc, Version: Latest, Offset: 2
2026.05.06 08:48:31.452667 [ 31633 ] {} <Debug> S3ObjectStorage: Objects with paths [test_02944/qwm/xryamieayuacqartbhkkzsuffepfc] were removed from S3
2026.05.06 08:48:31.452753 [ 31633 ] {} <Information> DiskObjectStorage(s3_disk_02944): Starting up disk s3_disk_02944
2026.05.06 08:48:31.452813 [ 31633 ] {} <Information> DiskObjectStorage(s3_disk_02944): Disk s3_disk_02944 started up
2026.05.06 08:48:31.452921 [ 31633 ] {} <Information> KeeperContext: Disk type 'cache' is not supported for Keeper
2026.05.06 08:48:31.458009 [ 31633 ] {} <Information> KeeperContext: Disk type 'object_storage' is not supported for Keeper
2026.05.06 08:48:31.460160 [ 31633 ] {} <Trace> KeeperSnapshotManager: Reading from disk LocalSnapshotDisk
2026.05.06 08:48:31.460331 [ 31633 ] {} <Trace> KeeperSnapshotManager: No snapshots were found on LocalSnapshotDisk
2026.05.06 08:48:31.461035 [ 31633 ] {} <Trace> KeeperLogStore: Reading from disk LocalLogDisk
2026.05.06 08:48:31.461301 [ 31633 ] {} <Trace> KeeperLogStore: Found changelog_1_100000.bin on LocalLogDisk
2026.05.06 08:48:31.461991 [ 31633 ] {} <Information> KeeperLogStore: force_sync disabled
2026.05.06 08:48:31.462210 [ 31633 ] {} <Debug> KeeperDispatcher: Waiting server to initialize
2026.05.06 08:48:31.462472 [ 31633 ] {} <Debug> KeeperStateMachine: Totally have 0 snapshots
2026.05.06 08:48:31.462627 [ 31633 ] {} <Debug> KeeperStateMachine: Trying to load state machine from snapshot up to log index 0
2026.05.06 08:48:31.462732 [ 31633 ] {} <Debug> KeeperStateMachine: No existing snapshots, last committed log index 0
2026.05.06 08:48:31.493450 [ 31633 ] {} <Trace> KeeperLogStore: Totally read from changelog changelog_1_100000.bin 3335 entries
2026.05.06 08:48:31.493622 [ 31633 ] {} <Trace> KeeperLogStore: Continue to write into changelog_1_100000.bin
2026.05.06 08:48:31.494694 [ 31633 ] {} <Trace> KeeperServer: Last local log idx 3335
2026.05.06 08:48:31.494773 [ 31633 ] {} <Information> KeeperServer: No config in snapshot, will use config from log store with log index 23
2026.05.06 08:48:31.495462 [ 31633 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2026.05.06 08:48:31.504949 [ 31633 ] {} <Information> RaftInstance: Raft ASIO listener initiated on :::9234, unsecured
2026.05.06 08:48:31.505167 [ 31633 ] {} <Information> KeeperStateManager: Read state from /var/lib/clickhouse/coordination/state
2026.05.06 08:48:31.505317 [ 31633 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2026.05.06 08:48:31.505407 [ 31633 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2026.05.06 08:48:31.505496 [ 31633 ] {} <Information> RaftInstance: parameters: election timeout range 0 - 0, heartbeat 0, leadership expiry 0, max batch 100, backoff 50, snapshot distance 10000, enable randomized snapshot creation NO, log sync stop gap 99999, reserved logs 2147483647, client timeout 10000, auto forwarding on, API call type async, custom commit quorum size 0, custom election quorum size 0, snapshot receiver included, leadership transfer wait time 0, grace period of lagging state machine 0, snapshot IO: blocking, parallel log appending: on
2026.05.06 08:48:31.505563 [ 31633 ] {} <Information> RaftInstance: new election timeout range: 0 - 0
2026.05.06 08:48:31.505678 [ 31633 ] {} <Information> RaftInstance:    === INIT RAFT SERVER ===
commit index 0
term 3
election timer allowed
log store start 1, end 3335
config log idx 23, prev log idx 10
 -- ASYNC REPLICATION --
2026.05.06 08:48:31.505737 [ 31633 ] {} <Information> RaftInstance: detect a configuration change that is not committed yet at index 1
2026.05.06 08:48:31.505848 [ 31633 ] {} <Information> RaftInstance: peer 1: DC ID 0, localhost:9234, voting member, 1
my id: 1, voting_member
num peers: 0
2026.05.06 08:48:31.505939 [ 31633 ] {} <Information> RaftInstance: global manager does not exist. will use local thread for commit and append
2026.05.06 08:48:31.506269 [ 31633 ] {} <Information> RaftInstance: wait for HB, for 50 + [0, 0] ms
2026.05.06 08:48:31.506883 [ 32261 ] {} <Information> RaftInstance: bg append_entries thread initiated
2026.05.06 08:48:31.556872 [ 32245 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2026.05.06 08:48:31.557178 [ 32245 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
2026.05.06 08:48:31.557199 [ 31633 ] {} <Debug> KeeperDispatcher: Server initialized, waiting for quorum
2026.05.06 08:48:31.557544 [ 32245 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: follower, log last term 3, state term 3, target p 1, my p 1, hb dead, pre-vote NOT done
2026.05.06 08:48:31.557694 [ 32245 ] {} <Information> RaftInstance: pre-vote term (0) is different, reset it to 3
2026.05.06 08:48:31.560424 [ 32245 ] {} <Debug> IDisk: Copying from LocalStateFileDisk (path: /var/lib/clickhouse/coordination) state to LocalStateFileDisk (path: /var/lib/clickhouse/coordination) state-OLD.
2026.05.06 08:48:31.570144 [ 32245 ] {} <Information> RaftInstance: [VOTE INIT] my id 1, my role candidate, term 4, log idx 3335, log term 3, priority (target 1 / mine 1)
2026.05.06 08:48:31.570291 [ 32245 ] {} <Information> RaftInstance: number of pending commit elements: 0
2026.05.06 08:48:31.570395 [ 32245 ] {} <Information> RaftInstance: state machine commit index 0, precommit index 3335, last log index 3335
2026.05.06 08:48:31.570615 [ 32245 ] {} <Information> RaftInstance: Currently assigned config is newer than some uncomitted config. This can only happen during startup or force recovery. If that is not the case, this is a bug.
2026.05.06 08:48:31.571060 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:31.571315 [ 32245 ] {} <Information> RaftInstance: [BECOME LEADER] appended new config at 3336
2026.05.06 08:48:31.571889 [ 32262 ] {} <Information> RaftInstance: config at index 1 is committed, prev config log idx 23
2026.05.06 08:48:31.572068 [ 32262 ] {} <Information> RaftInstance: skipped config 1, latest config 23
2026.05.06 08:48:31.572519 [ 32262 ] {} <Debug> KeeperStateMachine: Session ID response 1 with timeout 30000
2026.05.06 08:48:31.574850 [ 32262 ] {} <Information> RaftInstance: config at index 10 is committed, prev config log idx 23
2026.05.06 08:48:31.574923 [ 32262 ] {} <Information> RaftInstance: skipped config 10, latest config 23
2026.05.06 08:48:31.575045 [ 32262 ] {} <Debug> KeeperStateMachine: Session ID response 2 with timeout 30000
2026.05.06 08:48:31.576069 [ 32262 ] {} <Information> RaftInstance: config at index 23 is committed, prev config log idx 23
2026.05.06 08:48:31.576129 [ 32262 ] {} <Information> RaftInstance: skipped config 23, latest config 23
2026.05.06 08:48:31.576236 [ 32262 ] {} <Debug> KeeperStateMachine: Session ID response 3 with timeout 30000
2026.05.06 08:48:32.081433 [ 32262 ] {} <Debug> KeeperStateMachine: Session ID response 4 with timeout 30000
2026.05.06 08:48:32.086718 [ 32262 ] {} <Information> KeeperServer: Preprocessing 1 log entries
2026.05.06 08:48:32.086778 [ 32262 ] {} <Information> KeeperServer: Preprocessing done
2026.05.06 08:48:32.086841 [ 32262 ] {} <Information> RaftInstance: config at index 3336 is committed, prev config log idx 23
2026.05.06 08:48:32.086896 [ 32262 ] {} <Information> RaftInstance: new config log idx 3336, prev log idx 23, cur config log idx 23, prev log idx 10
2026.05.06 08:48:32.086970 [ 32262 ] {} <Information> RaftInstance: new configuration: log idx 3336, prev log idx 23
peer 1, DC ID 0, localhost:9234, voting member, 1
my id: 1, leader: 1, term: 4
2026.05.06 08:48:32.087227 [ 31633 ] {} <Debug> KeeperDispatcher: Quorum initialized
2026.05.06 08:48:32.088149 [ 31633 ] {} <Debug> KeeperDispatcher: Dispatcher initialized
2026.05.06 08:48:32.090742 [ 31633 ] {} <Information> Application: Listening for Keeper (tcp): [::]:9181
2026.05.06 08:48:32.091097 [ 31633 ] {} <Information> Application: Listening for replica communication (interserver): http://[::]:9009
2026.05.06 08:48:32.113009 [ 31633 ] {} <Debug> ConfigReloader: Loading config '/etc/clickhouse-server/users.xml'
2026.05.06 08:48:32.113190 [ 31633 ] {} <Debug> ConfigProcessor: Processing configuration file '/etc/clickhouse-server/users.xml'.
2026.05.06 08:48:32.115388 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/access_management.xml'.
2026.05.06 08:48:32.116664 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/allow_introspection_functions.yaml'.
2026.05.06 08:48:32.118328 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/database_atomic_drop_detach_sync.xml'.
2026.05.06 08:48:32.118948 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/database_ordinary.xml'.
2026.05.06 08:48:32.119469 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/enable_blobs_check.xml'.
2026.05.06 08:48:32.120041 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/filelog.xml'.
2026.05.06 08:48:32.120607 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/insert_keeper_retries.xml'.
2026.05.06 08:48:32.121401 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/log_queries.xml'.
2026.05.06 08:48:32.122013 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/marks.xml'.
2026.05.06 08:48:32.122613 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/memory_profiler.xml'.
2026.05.06 08:48:32.123278 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/no_fsync_metadata.xml'.
2026.05.06 08:48:32.124004 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/nonconst_timezone.xml'.
2026.05.06 08:48:32.124685 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/opentelemetry.xml'.
2026.05.06 08:48:32.125318 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/prefetch_settings.xml'.
2026.05.06 08:48:32.126170 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/readonly.xml'.
2026.05.06 08:48:32.127456 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/remote_queries.xml'.
2026.05.06 08:48:32.128325 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/replicated_ddl_entry.xml'.
2026.05.06 08:48:32.129068 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/s3_cache.xml'.
2026.05.06 08:48:32.129799 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/s3_cache_new.xml'.
2026.05.06 08:48:32.130421 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/session_log_test.xml'.
2026.05.06 08:48:32.131687 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/stress_test_tweaks-users.xml'.
2026.05.06 08:48:32.133585 [ 31633 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/timeouts.xml'.
2026.05.06 08:48:32.147715 [ 31633 ] {} <Debug> ConfigProcessor: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.
2026.05.06 08:48:32.148328 [ 31633 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/users.xml', performing update on configuration
2026.05.06 08:48:32.153385 [ 31633 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/users.xml', performed update on configuration
2026.05.06 08:48:32.153501 [ 31633 ] {} <Trace> ConfigReloader: Config reload interval set to 2000ms
2026.05.06 08:48:32.172220 [ 31633 ] {} <Debug> Access(user directories): Added users_xml access storage 'users_xml', path: /etc/clickhouse-server/users.xml
2026.05.06 08:48:32.177258 [ 31633 ] {} <Debug> Access(user directories): Added local_directory access storage 'local_directory', path: /var/lib/clickhouse/access/
2026.05.06 08:48:32.177606 [ 31633 ] {} <Information> CgroupsMemoryUsageObserver: Started cgroup current memory usage observer thread
2026.05.06 08:48:32.178890 [ 32270 ] {} <Information> CgroupsMemoryUsageObserver: Memory amount initially available to the process is 30.59 GiB
2026.05.06 08:48:32.185707 [ 31633 ] {} <Information> Context: Initialized background executor for merges and mutations with num_threads=16, num_tasks=32, scheduling_policy=round_robin
2026.05.06 08:48:32.189295 [ 31633 ] {} <Information> Context: Initialized background executor for move operations with num_threads=8, num_tasks=8
2026.05.06 08:48:32.196841 [ 31633 ] {} <Information> Context: Initialized background executor for fetches with num_threads=16, num_tasks=16
2026.05.06 08:48:32.200016 [ 31633 ] {} <Information> Context: Initialized background executor for common operations (e.g. clearing old parts) with num_threads=8, num_tasks=8
2026.05.06 08:48:32.208405 [ 31633 ] {} <Information> DNSCacheUpdater: Update period 15 seconds
2026.05.06 08:48:32.208575 [ 31633 ] {} <Information> Application: Loading metadata from /var/lib/clickhouse/
2026.05.06 08:48:32.208600 [ 31684 ] {} <Debug> DNSResolver: Updating DNS cache
2026.05.06 08:48:32.209323 [ 31684 ] {} <Debug> DNSResolver: Updated DNS cache
2026.05.06 08:48:32.238187 [ 32337 ] {} <Test> DiskObjectStorage(azure): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:32.238356 [ 32337 ] {} <Test> AzureObjectStorage: Writing file: leubmbxpprqwbqektdvesxrjwvnzybns
2026.05.06 08:48:32.252813 [ 32337 ] {} <Trace> WriteBufferFromAzureBlobStorage: Committed single block for blob `leubmbxpprqwbqektdvesxrjwvnzybns`
2026.05.06 08:48:32.253325 [ 32337 ] {} <Trace> WriteBufferFromAzureBlobStorage: finalizeImpl WriteBufferFromAzureBlobStorage leubmbxpprqwbqektdvesxrjwvnzybns
2026.05.06 08:48:32.256209 [ 32337 ] {} <Trace> WriteBufferFromAzureBlobStorage: Close WriteBufferFromAzureBlobStorage. leubmbxpprqwbqektdvesxrjwvnzybns.
2026.05.06 08:48:32.258684 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: leubmbxpprqwbqektdvesxrjwvnzybns (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.265850 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: leubmbxpprqwbqektdvesxrjwvnzybns (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.272144 [ 32337 ] {} <Test> AzureObjectStorage: Removing single object: leubmbxpprqwbqektdvesxrjwvnzybns
2026.05.06 08:48:32.275250 [ 32337 ] {} <Information> DiskObjectStorage(azure): Starting up disk azure
2026.05.06 08:48:32.275401 [ 32337 ] {} <Information> DiskObjectStorage(azure): Disk azure started up
2026.05.06 08:48:32.277522 [ 32337 ] {} <Debug> FileCache(cached_azure): Using LRU cache policy
2026.05.06 08:48:32.278047 [ 32337 ] {} <Information> FileCache(cached_azure): Loading filesystem cache with 16 threads from /var/lib/clickhouse/filesystem_caches/azure_cache/
2026.05.06 08:48:32.282166 [ 32337 ] {} <Information> StatusFile: Status file /var/lib/clickhouse/filesystem_caches/azure_cache/status already exists - unclean restart. Contents:
PID: 2722
Started at: 2026-05-06 07:51:04
Revision: 4294967295

2026.05.06 08:48:32.282622 [ 32337 ] {} <Information> DiskCache: Registered cached disk (`cached_azure`) with structure: DiskObjectStorage-azure(CachedObjectStorage-cached_azure(AzureObjectStorage))
2026.05.06 08:48:32.292103 [ 32337 ] {} <Information> AWSClient: ClientConfiguration: User agent is overridden in the config: ClickHouse 24.8.14.10546.altinitytest (altinity build)
2026.05.06 08:48:32.307096 [ 32337 ] {} <Trace> S3Client: Provider type: Unknown
2026.05.06 08:48:32.307169 [ 32337 ] {} <Trace> S3Client: API mode of the S3 client: AWS
2026.05.06 08:48:32.331089 [ 32337 ] {} <Debug> S3ObjectStorage: Objects with paths [s3/afi/uadsfvabwmqpbkrtgjtdbpkntgtvx] were removed from S3
2026.05.06 08:48:32.331595 [ 32337 ] {} <Test> DiskObjectStorage(s3_disk): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:32.355295 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vwi/zrccpbuxpvbnobiqfmcgaeqvtwylm (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.359775 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vwi/zrccpbuxpvbnobiqfmcgaeqvtwylm (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.359848 [ 32337 ] {} <Test> ReadBufferFromS3: Read S3 object. Bucket: test, Key: s3/vwi/zrccpbuxpvbnobiqfmcgaeqvtwylm, Version: Latest, Offset: 2
2026.05.06 08:48:32.365470 [ 32337 ] {} <Debug> S3ObjectStorage: Objects with paths [s3/vwi/zrccpbuxpvbnobiqfmcgaeqvtwylm] were removed from S3
2026.05.06 08:48:32.365569 [ 32337 ] {} <Information> DiskObjectStorage(s3_disk): Starting up disk s3_disk
2026.05.06 08:48:32.365614 [ 32337 ] {} <Information> DiskObjectStorage(s3_disk): Disk s3_disk started up
2026.05.06 08:48:32.367523 [ 32337 ] {} <Information> AWSClient: ClientConfiguration: User agent is overridden in the config: ClickHouse 24.8.14.10546.altinitytest (altinity build)
2026.05.06 08:48:32.374384 [ 32337 ] {} <Trace> S3Client: Provider type: Unknown
2026.05.06 08:48:32.374449 [ 32337 ] {} <Trace> S3Client: API mode of the S3 client: AWS
2026.05.06 08:48:32.391862 [ 32337 ] {} <Debug> S3ObjectStorage: Objects with paths [s3_plain/clickhouse_remove_objects_capability_11a20655-72d2-4e0e-b157-3c8461d3c947] were removed from S3
2026.05.06 08:48:32.392104 [ 32337 ] {} <Test> DiskObjectStorage(s3_plain_disk): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:32.408345 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3_plain/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947 (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.413637 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3_plain/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947 (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.413703 [ 32337 ] {} <Test> ReadBufferFromS3: Read S3 object. Bucket: test, Key: s3_plain/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947, Version: Latest, Offset: 2
2026.05.06 08:48:32.427215 [ 32337 ] {} <Debug> S3ObjectStorage: Objects with paths [s3_plain/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947] were removed from S3
2026.05.06 08:48:32.427282 [ 32337 ] {} <Information> DiskObjectStorage(s3_plain_disk): Starting up disk s3_plain_disk
2026.05.06 08:48:32.427319 [ 32337 ] {} <Information> DiskObjectStorage(s3_plain_disk): Disk s3_plain_disk started up
2026.05.06 08:48:32.428034 [ 32337 ] {} <Debug> SLRUFileCachePriority(s3_cache): Probationary queue 73400320 in size and 7000000 in elements. Protected queue 31457280 in size and 3000000 in elements
2026.05.06 08:48:32.428094 [ 32337 ] {} <Debug> FileCache(s3_cache): Using SLRU cache policy
2026.05.06 08:48:32.428283 [ 32337 ] {} <Information> FileCache(s3_cache): Loading filesystem cache with 16 threads from /var/lib/clickhouse/filesystem_caches/s3_cache/
2026.05.06 08:48:32.430937 [ 32347 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 67, current is 0
2026.05.06 08:48:32.431488 [ 32347 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: c19724c2f984163665bde94656daf920, offset: 0, size: 67
2026.05.06 08:48:32.431704 [ 32348 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 30, current is 67
2026.05.06 08:48:32.431956 [ 32348 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: d5797c02b9d7c7b8dd972216e219ff9d, offset: 0, size: 30
2026.05.06 08:48:32.432204 [ 32349 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 1, current is 97
2026.05.06 08:48:32.432374 [ 32348 ] {} <Test> FileCache(s3_cache): Added file segment d5797c02b9d7c7b8dd972216e219ff9d:0 (size: 30) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/d57/d5797c02b9d7c7b8dd972216e219ff9d/0
2026.05.06 08:48:32.432179 [ 32347 ] {} <Test> FileCache(s3_cache): Added file segment c19724c2f984163665bde94656daf920:0 (size: 67) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/c19/c19724c2f984163665bde94656daf920/0
2026.05.06 08:48:32.432354 [ 32349 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: eb35a528d8f8ea53396c56179f8107bc, offset: 0, size: 1
2026.05.06 08:48:32.432671 [ 32351 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 42, current is 98
2026.05.06 08:48:32.432786 [ 32349 ] {} <Test> FileCache(s3_cache): Added file segment eb35a528d8f8ea53396c56179f8107bc:0 (size: 1) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/eb3/eb35a528d8f8ea53396c56179f8107bc/0
2026.05.06 08:48:32.433240 [ 32351 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 43e8c82ab0d38793febc4fd366afc572, offset: 0, size: 42
2026.05.06 08:48:32.433332 [ 32348 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 1, current is 140
2026.05.06 08:48:32.433367 [ 32348 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: c398d2bbee132ab3d078997de3f82f98, offset: 0, size: 1
2026.05.06 08:48:32.433417 [ 32347 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 10, current is 141
2026.05.06 08:48:32.433444 [ 32348 ] {} <Test> FileCache(s3_cache): Added file segment c398d2bbee132ab3d078997de3f82f98:0 (size: 1) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/c39/c398d2bbee132ab3d078997de3f82f98/0
2026.05.06 08:48:32.433456 [ 32347 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 786f9b3fa20c683a39db0316dde3086f, offset: 0, size: 10
2026.05.06 08:48:32.433538 [ 32347 ] {} <Test> FileCache(s3_cache): Added file segment 786f9b3fa20c683a39db0316dde3086f:0 (size: 10) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/786/786f9b3fa20c683a39db0316dde3086f/0
2026.05.06 08:48:32.433525 [ 32351 ] {} <Test> FileCache(s3_cache): Added file segment 43e8c82ab0d38793febc4fd366afc572:0 (size: 42) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/43e/43e8c82ab0d38793febc4fd366afc572/0
2026.05.06 08:48:32.433578 [ 32353 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 276, current is 151
2026.05.06 08:48:32.433994 [ 32353 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 996dedb7dc3c5a127eefbc86122b0cfc, offset: 0, size: 276
2026.05.06 08:48:32.434437 [ 32354 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 47, current is 427
2026.05.06 08:48:32.434647 [ 32353 ] {} <Test> FileCache(s3_cache): Added file segment 996dedb7dc3c5a127eefbc86122b0cfc:0 (size: 276) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/996/996dedb7dc3c5a127eefbc86122b0cfc/0
2026.05.06 08:48:32.435202 [ 32354 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 243055bd05e5edb7312d2eba2b908091, offset: 0, size: 47
2026.05.06 08:48:32.435326 [ 32355 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 10, current is 474
2026.05.06 08:48:32.435658 [ 32355 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 1f29f94719cc1b8336e4b3a854c89221, offset: 0, size: 10
2026.05.06 08:48:32.435804 [ 32349 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 30, current is 484
2026.05.06 08:48:32.435902 [ 32349 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: f7384aae36fe4289de559f96d819c565, offset: 0, size: 30
2026.05.06 08:48:32.435981 [ 32348 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 30, current is 514
2026.05.06 08:48:32.436019 [ 32348 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: ea3aacb6887d7efc651b45dac4557386, offset: 0, size: 30
2026.05.06 08:48:32.435671 [ 32354 ] {} <Test> FileCache(s3_cache): Added file segment 243055bd05e5edb7312d2eba2b908091:0 (size: 47) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/243/243055bd05e5edb7312d2eba2b908091/0
2026.05.06 08:48:32.436072 [ 32349 ] {} <Test> FileCache(s3_cache): Added file segment f7384aae36fe4289de559f96d819c565:0 (size: 30) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/f73/f7384aae36fe4289de559f96d819c565/0
2026.05.06 08:48:32.436087 [ 32348 ] {} <Test> FileCache(s3_cache): Added file segment ea3aacb6887d7efc651b45dac4557386:0 (size: 30) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/ea3/ea3aacb6887d7efc651b45dac4557386/0
2026.05.06 08:48:32.436120 [ 32355 ] {} <Test> FileCache(s3_cache): Added file segment 1f29f94719cc1b8336e4b3a854c89221:0 (size: 10) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/1f2/1f29f94719cc1b8336e4b3a854c89221/0
2026.05.06 08:48:32.436104 [ 32356 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 47, current is 544
2026.05.06 08:48:32.436545 [ 32356 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 33385792238d709413a230095c6af1d8, offset: 0, size: 47
2026.05.06 08:48:32.436655 [ 32347 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 205, current is 591
2026.05.06 08:48:32.436748 [ 32347 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 138075dc4f908b8696832089aba07a75, offset: 0, size: 205
2026.05.06 08:48:32.436763 [ 32356 ] {} <Test> FileCache(s3_cache): Added file segment 33385792238d709413a230095c6af1d8:0 (size: 47) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/333/33385792238d709413a230095c6af1d8/0
2026.05.06 08:48:32.436820 [ 32348 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 30, current is 796
2026.05.06 08:48:32.436852 [ 32348 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 19c6e9169e44018ecf16bae502ce1ca2, offset: 0, size: 30
2026.05.06 08:48:32.436912 [ 32348 ] {} <Test> FileCache(s3_cache): Added file segment 19c6e9169e44018ecf16bae502ce1ca2:0 (size: 30) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/19c/19c6e9169e44018ecf16bae502ce1ca2/0
2026.05.06 08:48:32.436911 [ 32347 ] {} <Test> FileCache(s3_cache): Added file segment 138075dc4f908b8696832089aba07a75:0 (size: 205) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/138/138075dc4f908b8696832089aba07a75/0
2026.05.06 08:48:32.436926 [ 32354 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 276, current is 826
2026.05.06 08:48:32.437003 [ 32354 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 3606a251266710571c29f7d537f11077, offset: 0, size: 276
2026.05.06 08:48:32.437101 [ 32355 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 42, current is 1102
2026.05.06 08:48:32.437150 [ 32354 ] {} <Test> FileCache(s3_cache): Added file segment 3606a251266710571c29f7d537f11077:0 (size: 276) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/360/3606a251266710571c29f7d537f11077/0
2026.05.06 08:48:32.437174 [ 32355 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: c5c9ecce30a515cdbe1f1fd129013899, offset: 0, size: 42
2026.05.06 08:48:32.437281 [ 32349 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 276, current is 1144
2026.05.06 08:48:32.437321 [ 32355 ] {} <Test> FileCache(s3_cache): Added file segment c5c9ecce30a515cdbe1f1fd129013899:0 (size: 42) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/c5c/c5c9ecce30a515cdbe1f1fd129013899/0
2026.05.06 08:48:32.437359 [ 32349 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 349af8778c2543613ccc71c8f0f56bd8, offset: 0, size: 276
2026.05.06 08:48:32.437439 [ 32354 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 47, current is 1420
2026.05.06 08:48:32.437519 [ 32354 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 5f5b22d53cd0bf5e89bacec301f68529, offset: 0, size: 47
2026.05.06 08:48:32.437521 [ 32349 ] {} <Test> FileCache(s3_cache): Added file segment 349af8778c2543613ccc71c8f0f56bd8:0 (size: 276) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/349/349af8778c2543613ccc71c8f0f56bd8/0
2026.05.06 08:48:32.437597 [ 32348 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 1, current is 1467
2026.05.06 08:48:32.437629 [ 32348 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 98e8bed7d70180dd805707a7b9c12d5f, offset: 0, size: 1
2026.05.06 08:48:32.437665 [ 32354 ] {} <Test> FileCache(s3_cache): Added file segment 5f5b22d53cd0bf5e89bacec301f68529:0 (size: 47) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/5f5/5f5b22d53cd0bf5e89bacec301f68529/0
2026.05.06 08:48:32.437693 [ 32348 ] {} <Test> FileCache(s3_cache): Added file segment 98e8bed7d70180dd805707a7b9c12d5f:0 (size: 1) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/98e/98e8bed7d70180dd805707a7b9c12d5f/0
2026.05.06 08:48:32.437707 [ 32347 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 30, current is 1468
2026.05.06 08:48:32.437781 [ 32347 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: c07fcbbc467687c012018cb66b7403b6, offset: 0, size: 30
2026.05.06 08:48:32.437871 [ 32356 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 1, current is 1498
2026.05.06 08:48:32.437933 [ 32356 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: e975169734962c94d801165a31501fba, offset: 0, size: 1
2026.05.06 08:48:32.437929 [ 32347 ] {} <Test> FileCache(s3_cache): Added file segment c07fcbbc467687c012018cb66b7403b6:0 (size: 30) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/c07/c07fcbbc467687c012018cb66b7403b6/0
2026.05.06 08:48:32.438053 [ 32356 ] {} <Test> FileCache(s3_cache): Added file segment e975169734962c94d801165a31501fba:0 (size: 1) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/e97/e975169734962c94d801165a31501fba/0
2026.05.06 08:48:32.438184 [ 32347 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 276, current is 1499
2026.05.06 08:48:32.438253 [ 32347 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: c07538018b3df63ed790dfe1d5992011, offset: 0, size: 276
2026.05.06 08:48:32.438384 [ 32347 ] {} <Test> FileCache(s3_cache): Added file segment c07538018b3df63ed790dfe1d5992011:0 (size: 276) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/c07/c07538018b3df63ed790dfe1d5992011/0
2026.05.06 08:48:32.472001 [ 32353 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 1, current is 1775
2026.05.06 08:48:32.472097 [ 32353 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: d96b10f818af9fc56b70c3886e85f9c3, offset: 0, size: 1
2026.05.06 08:48:32.472230 [ 32351 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 47, current is 1776
2026.05.06 08:48:32.472266 [ 32353 ] {} <Test> FileCache(s3_cache): Added file segment d96b10f818af9fc56b70c3886e85f9c3:0 (size: 1) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/d96/d96b10f818af9fc56b70c3886e85f9c3/0
2026.05.06 08:48:32.472326 [ 32351 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 4a64d929d163d7f1862333141ff291c6, offset: 0, size: 47
2026.05.06 08:48:32.472446 [ 32363 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 276, current is 1823
2026.05.06 08:48:32.472509 [ 32351 ] {} <Test> FileCache(s3_cache): Added file segment 4a64d929d163d7f1862333141ff291c6:0 (size: 47) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/4a6/4a64d929d163d7f1862333141ff291c6/0
2026.05.06 08:48:32.472618 [ 32363 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 746ffd5a9ae9d2c155b605c87af24d40, offset: 0, size: 276
2026.05.06 08:48:32.472791 [ 32363 ] {} <Test> FileCache(s3_cache): Added file segment 746ffd5a9ae9d2c155b605c87af24d40:0 (size: 276) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/746/746ffd5a9ae9d2c155b605c87af24d40/0
2026.05.06 08:48:32.472767 [ 32358 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 47, current is 2099
2026.05.06 08:48:32.472967 [ 32358 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 3244d0979a74840152a85a31e615952a, offset: 0, size: 47
2026.05.06 08:48:32.473097 [ 32360 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 276, current is 2146
2026.05.06 08:48:32.473222 [ 32360 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 487e7bf3b6be9c77e51360e1bb044e38, offset: 0, size: 276
2026.05.06 08:48:32.473248 [ 32358 ] {} <Test> FileCache(s3_cache): Added file segment 3244d0979a74840152a85a31e615952a:0 (size: 47) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/324/3244d0979a74840152a85a31e615952a/0
2026.05.06 08:48:32.473356 [ 32357 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 205, current is 2422
2026.05.06 08:48:32.473448 [ 32360 ] {} <Test> FileCache(s3_cache): Added file segment 487e7bf3b6be9c77e51360e1bb044e38:0 (size: 276) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/487/487e7bf3b6be9c77e51360e1bb044e38/0
2026.05.06 08:48:32.473530 [ 32357 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 0f8745be7660054901be780588901782, offset: 0, size: 205
2026.05.06 08:48:32.473652 [ 32365 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 47, current is 2627
2026.05.06 08:48:32.473726 [ 32357 ] {} <Test> FileCache(s3_cache): Added file segment 0f8745be7660054901be780588901782:0 (size: 205) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/0f8/0f8745be7660054901be780588901782/0
2026.05.06 08:48:32.473783 [ 32365 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 9aefc7ccc02ad835a2557bfa2eacdfd4, offset: 0, size: 47
2026.05.06 08:48:32.473909 [ 32362 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 1, current is 2674
2026.05.06 08:48:32.473965 [ 32365 ] {} <Test> FileCache(s3_cache): Added file segment 9aefc7ccc02ad835a2557bfa2eacdfd4:0 (size: 47) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/9ae/9aefc7ccc02ad835a2557bfa2eacdfd4/0
2026.05.06 08:48:32.474027 [ 32362 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: f069604742ee525a04e9e016584d3212, offset: 0, size: 1
2026.05.06 08:48:32.474175 [ 32361 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 30, current is 2675
2026.05.06 08:48:32.474207 [ 32362 ] {} <Test> FileCache(s3_cache): Added file segment f069604742ee525a04e9e016584d3212:0 (size: 1) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/f06/f069604742ee525a04e9e016584d3212/0
2026.05.06 08:48:32.474343 [ 32361 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: b45fb6ca9b2f8a5ca572317d7c145f84, offset: 0, size: 30
2026.05.06 08:48:32.474546 [ 32359 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with 67, current is 2705
2026.05.06 08:48:32.474576 [ 32361 ] {} <Test> FileCache(s3_cache): Added file segment b45fb6ca9b2f8a5ca572317d7c145f84:0 (size: 30) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/b45/b45fb6ca9b2f8a5ca572317d7c145f84/0
2026.05.06 08:48:32.474734 [ 32359 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Added entry into LRU queue, key: 4f35d3dc379904cf70c62240c71a85ac, offset: 0, size: 67
2026.05.06 08:48:32.474918 [ 32359 ] {} <Test> FileCache(s3_cache): Added file segment 4f35d3dc379904cf70c62240c71a85ac:0 (size: 67) with path: /var/lib/clickhouse/filesystem_caches/s3_cache/4f3/4f35d3dc379904cf70c62240c71a85ac/0
2026.05.06 08:48:32.477114 [ 32337 ] {} <Information> StatusFile: Status file /var/lib/clickhouse/filesystem_caches/s3_cache/status already exists - unclean restart. Contents:
PID: 2722
Started at: 2026-05-06 07:51:05
Revision: 4294967295

2026.05.06 08:48:32.477458 [ 32337 ] {} <Information> DiskCache: Registered cached disk (`s3_cache`) with structure: DiskObjectStorage-s3_disk(CachedObjectStorage-s3_cache(S3ObjectStorage))
2026.05.06 08:48:32.478988 [ 32337 ] {} <Debug> FileCache(s3_cache_02933): Using LRU cache policy
2026.05.06 08:48:32.479453 [ 32337 ] {} <Information> FileCache(s3_cache_02933): Loading filesystem cache with 16 threads from /var/lib/clickhouse/filesystem_caches/s3_cache_02933/
2026.05.06 08:48:32.481903 [ 32337 ] {} <Information> StatusFile: Status file /var/lib/clickhouse/filesystem_caches/s3_cache_02933/status already exists - unclean restart. Contents:
PID: 2722
Started at: 2026-05-06 07:51:05
Revision: 4294967295

2026.05.06 08:48:32.482169 [ 32337 ] {} <Information> DiskCache: Registered cached disk (`s3_cache_02933`) with structure: DiskObjectStorage-s3_disk(CachedObjectStorage-s3_cache_02933(S3ObjectStorage))
2026.05.06 08:48:32.483008 [ 32337 ] {} <Test> DiskObjectStorage(local_disk): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:32.483133 [ 32337 ] {} <Test> LocalObjectStorage: Write object: local_disk/ijthxncxflzcaevqaafociaswooubobq
2026.05.06 08:48:32.490318 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: local_disk/ijthxncxflzcaevqaafociaswooubobq (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.492758 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: local_disk/ijthxncxflzcaevqaafociaswooubobq (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.493856 [ 32337 ] {} <Information> DiskObjectStorage(local_disk): Starting up disk local_disk
2026.05.06 08:48:32.493928 [ 32337 ] {} <Information> DiskObjectStorage(local_disk): Disk local_disk started up
2026.05.06 08:48:32.494526 [ 32337 ] {} <Test> DiskObjectStorage(local_disk_2): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:32.494600 [ 32337 ] {} <Test> LocalObjectStorage: Write object: local_disk_2/rkuqbocdsuomuagvdvhgaygvrhfgtnfi
2026.05.06 08:48:32.499276 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: local_disk_2/rkuqbocdsuomuagvdvhgaygvrhfgtnfi (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.500465 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: local_disk_2/rkuqbocdsuomuagvdvhgaygvrhfgtnfi (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.501005 [ 32337 ] {} <Information> DiskObjectStorage(local_disk_2): Starting up disk local_disk_2
2026.05.06 08:48:32.501038 [ 32337 ] {} <Information> DiskObjectStorage(local_disk_2): Disk local_disk_2 started up
2026.05.06 08:48:32.501345 [ 32337 ] {} <Test> DiskObjectStorage(local_disk_3): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:32.501382 [ 32337 ] {} <Test> LocalObjectStorage: Write object: local_disk_3/xsfwsxictdqaqcgerqqyuusfnvpszldu
2026.05.06 08:48:32.504118 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: local_disk_3/xsfwsxictdqaqcgerqqyuusfnvpszldu (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.505301 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: local_disk_3/xsfwsxictdqaqcgerqqyuusfnvpszldu (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.505852 [ 32337 ] {} <Information> DiskObjectStorage(local_disk_3): Starting up disk local_disk_3
2026.05.06 08:48:32.505884 [ 32337 ] {} <Information> DiskObjectStorage(local_disk_3): Disk local_disk_3 started up
2026.05.06 08:48:32.506434 [ 32337 ] {} <Debug> FileCache(local_cache): Using LRU cache policy
2026.05.06 08:48:32.506598 [ 32337 ] {} <Information> FileCache(local_cache): Loading filesystem cache with 16 threads from /var/lib/clickhouse/filesystem_caches/local_cache/
2026.05.06 08:48:32.507920 [ 32337 ] {} <Information> StatusFile: Status file /var/lib/clickhouse/filesystem_caches/local_cache/status already exists - unclean restart. Contents:
PID: 2722
Started at: 2026-05-06 07:51:05
Revision: 4294967295

2026.05.06 08:48:32.508041 [ 32337 ] {} <Information> DiskCache: Registered cached disk (`local_cache`) with structure: DiskObjectStorage-local_disk(CachedObjectStorage-local_cache(LocalObjectStorage))
2026.05.06 08:48:32.508565 [ 32337 ] {} <Debug> FileCache(local_cache_2): Using LRU cache policy
2026.05.06 08:48:32.508691 [ 32337 ] {} <Information> FileCache(local_cache_2): Loading filesystem cache with 16 threads from /var/lib/clickhouse/filesystem_caches/local_cache_2/
2026.05.06 08:48:32.509990 [ 32337 ] {} <Information> StatusFile: Status file /var/lib/clickhouse/filesystem_caches/local_cache_2/status already exists - unclean restart. Contents:
PID: 2722
Started at: 2026-05-06 07:51:05
Revision: 4294967295

2026.05.06 08:48:32.510093 [ 32337 ] {} <Information> DiskCache: Registered cached disk (`local_cache_2`) with structure: DiskObjectStorage-local_disk(CachedObjectStorage-local_cache_2(LocalObjectStorage))
2026.05.06 08:48:32.510621 [ 32337 ] {} <Debug> FileCache(local_cache_3): Using LRU cache policy
2026.05.06 08:48:32.510747 [ 32337 ] {} <Information> FileCache(local_cache_3): Loading filesystem cache with 16 threads from /var/lib/clickhouse/filesystem_caches/local_cache_3/
2026.05.06 08:48:32.511830 [ 32337 ] {} <Information> StatusFile: Status file /var/lib/clickhouse/filesystem_caches/local_cache_3/status already exists - unclean restart. Contents:
PID: 2722
Started at: 2026-05-06 07:51:05
Revision: 4294967295

2026.05.06 08:48:32.511944 [ 32337 ] {} <Information> DiskCache: Registered cached disk (`local_cache_3`) with structure: DiskObjectStorage-local_disk_3(CachedObjectStorage-local_cache_3(LocalObjectStorage))
2026.05.06 08:48:32.512454 [ 32337 ] {} <Debug> FileCache(s3_cache_multi): Using LRU cache policy
2026.05.06 08:48:32.512597 [ 32337 ] {} <Information> FileCache(s3_cache_multi): Loading filesystem cache with 16 threads from /var/lib/clickhouse/filesystem_caches/s3_cache_multi/
2026.05.06 08:48:32.513785 [ 32337 ] {} <Information> StatusFile: Status file /var/lib/clickhouse/filesystem_caches/s3_cache_multi/status already exists - unclean restart. Contents:
PID: 2722
Started at: 2026-05-06 07:51:05
Revision: 4294967295

2026.05.06 08:48:32.513886 [ 32337 ] {} <Information> DiskCache: Registered cached disk (`s3_cache_multi`) with structure: DiskObjectStorage-s3_disk(CachedObjectStorage-s3_cache_multi(CachedObjectStorage-s3_cache(S3ObjectStorage)))
2026.05.06 08:48:32.514396 [ 32337 ] {} <Debug> FileCache(s3_cache_multi_2): Using LRU cache policy
2026.05.06 08:48:32.514550 [ 32337 ] {} <Information> FileCache(s3_cache_multi_2): Loading filesystem cache with 16 threads from /var/lib/clickhouse/filesystem_caches/s3_cache_multi_2/
2026.05.06 08:48:32.515737 [ 32337 ] {} <Information> StatusFile: Status file /var/lib/clickhouse/filesystem_caches/s3_cache_multi_2/status already exists - unclean restart. Contents:
PID: 2722
Started at: 2026-05-06 07:51:05
Revision: 4294967295

2026.05.06 08:48:32.515847 [ 32337 ] {} <Information> DiskCache: Registered cached disk (`s3_cache_multi_2`) with structure: DiskObjectStorage-s3_disk(CachedObjectStorage-s3_cache_multi_2(CachedObjectStorage-s3_cache_multi(CachedObjectStorage-s3_cache(S3ObjectStorage))))
2026.05.06 08:48:32.517500 [ 32337 ] {} <Information> AWSClient: ClientConfiguration: User agent is overridden in the config: ClickHouse 24.8.14.10546.altinitytest (altinity build)
2026.05.06 08:48:32.524387 [ 32337 ] {} <Trace> S3Client: Provider type: Unknown
2026.05.06 08:48:32.524434 [ 32337 ] {} <Trace> S3Client: API mode of the S3 client: AWS
2026.05.06 08:48:32.537610 [ 32337 ] {} <Debug> S3ObjectStorage: Objects with paths [special/thf/arjbhmbngvdoegnlkxsosufymrein] were removed from S3
2026.05.06 08:48:32.537852 [ 32337 ] {} <Test> DiskObjectStorage(s3_no_cache): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:32.549305 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: special/rry/wztxnypnrkoyjxebidtsifwnzgfbw (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.552753 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: special/rry/wztxnypnrkoyjxebidtsifwnzgfbw (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.552821 [ 32337 ] {} <Test> ReadBufferFromS3: Read S3 object. Bucket: test, Key: special/rry/wztxnypnrkoyjxebidtsifwnzgfbw, Version: Latest, Offset: 2
2026.05.06 08:48:32.557990 [ 32337 ] {} <Debug> S3ObjectStorage: Objects with paths [special/rry/wztxnypnrkoyjxebidtsifwnzgfbw] were removed from S3
2026.05.06 08:48:32.558058 [ 32337 ] {} <Information> DiskObjectStorage(s3_no_cache): Starting up disk s3_no_cache
2026.05.06 08:48:32.558102 [ 32337 ] {} <Information> DiskObjectStorage(s3_no_cache): Disk s3_no_cache started up
2026.05.06 08:48:32.559733 [ 32337 ] {} <Information> AWSClient: ClientConfiguration: User agent is overridden in the config: ClickHouse 24.8.14.10546.altinitytest (altinity build)
2026.05.06 08:48:32.566526 [ 32337 ] {} <Trace> S3Client: Provider type: Unknown
2026.05.06 08:48:32.566570 [ 32337 ] {} <Trace> S3Client: API mode of the S3 client: AWS
2026.05.06 08:48:32.582654 [ 32337 ] {} <Debug> S3ObjectStorage: Objects with paths [test_02944/vov/hmsgsjndfnunpnzdplipwztxwfgof] were removed from S3
2026.05.06 08:48:32.582905 [ 32337 ] {} <Test> DiskObjectStorage(s3_disk_02944): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:32.597564 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: test_02944/dei/qzcuinuplpxhitrpxzgjvkzzoqqjf (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.600908 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: test_02944/dei/qzcuinuplpxhitrpxzgjvkzzoqqjf (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.600972 [ 32337 ] {} <Test> ReadBufferFromS3: Read S3 object. Bucket: test, Key: test_02944/dei/qzcuinuplpxhitrpxzgjvkzzoqqjf, Version: Latest, Offset: 2
2026.05.06 08:48:32.606333 [ 32337 ] {} <Debug> S3ObjectStorage: Objects with paths [test_02944/dei/qzcuinuplpxhitrpxzgjvkzzoqqjf] were removed from S3
2026.05.06 08:48:32.606398 [ 32337 ] {} <Information> DiskObjectStorage(s3_disk_02944): Starting up disk s3_disk_02944
2026.05.06 08:48:32.606434 [ 32337 ] {} <Information> DiskObjectStorage(s3_disk_02944): Disk s3_disk_02944 started up
2026.05.06 08:48:32.607061 [ 32337 ] {} <Debug> FileCache(s3_cache_02944): Using LRU cache policy
2026.05.06 08:48:32.607228 [ 32337 ] {} <Information> FileCache(s3_cache_02944): Loading filesystem cache with 16 threads from /var/lib/clickhouse/filesystem_caches/s3_cache_02944/
2026.05.06 08:48:32.608566 [ 32337 ] {} <Information> StatusFile: Status file /var/lib/clickhouse/filesystem_caches/s3_cache_02944/status already exists - unclean restart. Contents:
PID: 2722
Started at: 2026-05-06 07:51:05
Revision: 4294967295

2026.05.06 08:48:32.608692 [ 32337 ] {} <Information> DiskCache: Registered cached disk (`s3_cache_02944`) with structure: DiskObjectStorage-s3_disk(CachedObjectStorage-s3_cache_02944(S3ObjectStorage))
2026.05.06 08:48:32.615030 [ 32337 ] {} <Information> AWSClient: ClientConfiguration: User agent is overridden in the config: ClickHouse 24.8.14.10546.altinitytest (altinity build)
2026.05.06 08:48:32.622015 [ 32337 ] {} <Trace> S3Client: Provider type: Unknown
2026.05.06 08:48:32.622068 [ 32337 ] {} <Trace> S3Client: API mode of the S3 client: AWS
2026.05.06 08:48:32.638668 [ 32337 ] {} <Debug> S3ObjectStorage: Objects with paths [common/jud/esngycaxqoaggauewilmtuyxkibxl] were removed from S3
2026.05.06 08:48:32.638963 [ 32337 ] {} <Test> DiskObjectStorage(s3_disk_02963): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:32.653997 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: common/sxs/pchyqdsoundurpmasrkrcfgbcwcuj (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.657536 [ 32337 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: common/sxs/pchyqdsoundurpmasrkrcfgbcwcuj (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:32.657601 [ 32337 ] {} <Test> ReadBufferFromS3: Read S3 object. Bucket: test, Key: common/sxs/pchyqdsoundurpmasrkrcfgbcwcuj, Version: Latest, Offset: 2
2026.05.06 08:48:32.663016 [ 32337 ] {} <Debug> S3ObjectStorage: Objects with paths [common/sxs/pchyqdsoundurpmasrkrcfgbcwcuj] were removed from S3
2026.05.06 08:48:32.663081 [ 32337 ] {} <Information> DiskObjectStorage(s3_disk_02963): Starting up disk s3_disk_02963
2026.05.06 08:48:32.663119 [ 32337 ] {} <Information> DiskObjectStorage(s3_disk_02963): Disk s3_disk_02963 started up
2026.05.06 08:48:32.663568 [ 32337 ] {} <Trace> StoragePolicy (azure_cache): Storage policy azure_cache created, total volumes 1
2026.05.06 08:48:32.663669 [ 32337 ] {} <Information> StoragePolicySelector: Storage policy `azure_cache` loaded
2026.05.06 08:48:32.664052 [ 32337 ] {} <Trace> StoragePolicy (local_remote): Storage policy local_remote created, total volumes 2
2026.05.06 08:48:32.664091 [ 32337 ] {} <Information> StoragePolicySelector: Storage policy `local_remote` loaded
2026.05.06 08:48:32.664273 [ 32337 ] {} <Trace> StoragePolicy (s3_cache): Storage policy s3_cache created, total volumes 1
2026.05.06 08:48:32.664316 [ 32337 ] {} <Information> StoragePolicySelector: Storage policy `s3_cache` loaded
2026.05.06 08:48:32.664504 [ 32337 ] {} <Trace> StoragePolicy (s3_no_cache): Storage policy s3_no_cache created, total volumes 1
2026.05.06 08:48:32.664533 [ 32337 ] {} <Information> StoragePolicySelector: Storage policy `s3_no_cache` loaded
2026.05.06 08:48:32.664711 [ 32337 ] {} <Trace> StoragePolicy (s3_cache_multi): Storage policy s3_cache_multi created, total volumes 1
2026.05.06 08:48:32.664742 [ 32337 ] {} <Information> StoragePolicySelector: Storage policy `s3_cache_multi` loaded
2026.05.06 08:48:32.664941 [ 32337 ] {} <Trace> StoragePolicy (local_cache): Storage policy local_cache created, total volumes 1
2026.05.06 08:48:32.664987 [ 32337 ] {} <Information> StoragePolicySelector: Storage policy `local_cache` loaded
2026.05.06 08:48:32.665173 [ 32337 ] {} <Trace> StoragePolicy (local_cache_2): Storage policy local_cache_2 created, total volumes 1
2026.05.06 08:48:32.665204 [ 32337 ] {} <Information> StoragePolicySelector: Storage policy `local_cache_2` loaded
2026.05.06 08:48:32.665385 [ 32337 ] {} <Trace> StoragePolicy (local_cache_3): Storage policy local_cache_3 created, total volumes 1
2026.05.06 08:48:32.665425 [ 32337 ] {} <Information> StoragePolicySelector: Storage policy `local_cache_3` loaded
2026.05.06 08:48:32.666149 [ 32337 ] {} <Trace> StoragePolicy (policy_02961): Storage policy policy_02961 created, total volumes 4
2026.05.06 08:48:32.666198 [ 32337 ] {} <Information> StoragePolicySelector: Storage policy `policy_02961` loaded
2026.05.06 08:48:32.666296 [ 32337 ] {} <Trace> StoragePolicy (default): Storage policy default created, total volumes 1
2026.05.06 08:48:32.822707 [ 31633 ] {} <Information> DatabaseAtomic (system): Metadata processed, database system has 20 tables and 0 dictionaries in total.
2026.05.06 08:48:32.822998 [ 31633 ] {} <Information> TablesLoader: Parsed metadata of 20 tables in 1 databases in 0.610678763 sec
2026.05.06 08:48:32.832104 [ 31633 ] {} <Trace> ReferentialDeps: No tables
2026.05.06 08:48:32.832386 [ 31633 ] {} <Trace> LoadingDeps: Table system.trace_log has no dependencies (level 0)
2026.05.06 08:48:32.832549 [ 31633 ] {} <Trace> LoadingDeps: Table system.transactions_info_log has no dependencies (level 0)
2026.05.06 08:48:32.832647 [ 31633 ] {} <Trace> LoadingDeps: Table system.s3queue_log has no dependencies (level 0)
2026.05.06 08:48:32.832760 [ 31633 ] {} <Trace> LoadingDeps: Table system.query_thread_log has no dependencies (level 0)
2026.05.06 08:48:32.832849 [ 31633 ] {} <Trace> LoadingDeps: Table system.session_log has no dependencies (level 0)
2026.05.06 08:48:32.832937 [ 31633 ] {} <Trace> LoadingDeps: Table system.part_log has no dependencies (level 0)
2026.05.06 08:48:32.833024 [ 31633 ] {} <Trace> LoadingDeps: Table system.backup_log has no dependencies (level 0)
2026.05.06 08:48:32.833116 [ 31633 ] {} <Trace> LoadingDeps: Table system.asynchronous_metric_log has no dependencies (level 0)
2026.05.06 08:48:32.833204 [ 31633 ] {} <Trace> LoadingDeps: Table system.error_log has no dependencies (level 0)
2026.05.06 08:48:32.833293 [ 31633 ] {} <Trace> LoadingDeps: Table system.filesystem_cache_log has no dependencies (level 0)
2026.05.06 08:48:32.833380 [ 31633 ] {} <Trace> LoadingDeps: Table system.text_log has no dependencies (level 0)
2026.05.06 08:48:32.833465 [ 31633 ] {} <Trace> LoadingDeps: Table system.crash_log has no dependencies (level 0)
2026.05.06 08:48:32.833585 [ 31633 ] {} <Trace> LoadingDeps: Table system.opentelemetry_span_log has no dependencies (level 0)
2026.05.06 08:48:32.833692 [ 31633 ] {} <Trace> LoadingDeps: Table system.blob_storage_log has no dependencies (level 0)
2026.05.06 08:48:32.833781 [ 31633 ] {} <Trace> LoadingDeps: Table system.zookeeper_log has no dependencies (level 0)
2026.05.06 08:48:32.833867 [ 31633 ] {} <Trace> LoadingDeps: Table system.query_views_log has no dependencies (level 0)
2026.05.06 08:48:32.833954 [ 31633 ] {} <Trace> LoadingDeps: Table system.processors_profile_log has no dependencies (level 0)
2026.05.06 08:48:32.834041 [ 31633 ] {} <Trace> LoadingDeps: Table system.query_log has no dependencies (level 0)
2026.05.06 08:48:32.834147 [ 31633 ] {} <Trace> LoadingDeps: Table system.metric_log has no dependencies (level 0)
2026.05.06 08:48:32.834235 [ 31633 ] {} <Trace> LoadingDeps: Table system.asynchronous_insert_log has no dependencies (level 0)
2026.05.06 08:48:32.835568 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.trace_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.835758 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.transactions_info_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.835847 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.s3queue_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.835928 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.query_thread_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.835999 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.session_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836070 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.part_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836182 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.backup_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836256 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.asynchronous_metric_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836328 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.error_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836400 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.filesystem_cache_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836471 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.text_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836553 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.crash_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836635 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.opentelemetry_span_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836710 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.blob_storage_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836782 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.zookeeper_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836853 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.query_views_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836925 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.processors_profile_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.836995 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.query_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.837065 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.metric_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.837137 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'load table system.asynchronous_insert_log': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:32.837367 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.part_log' into ForegroundLoad
2026.05.06 08:48:32.837502 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.session_log' into ForegroundLoad
2026.05.06 08:48:32.837590 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.filesystem_cache_log' into ForegroundLoad
2026.05.06 08:48:32.837664 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.trace_log' into ForegroundLoad
2026.05.06 08:48:32.837731 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.query_views_log' into ForegroundLoad
2026.05.06 08:48:32.837799 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.query_thread_log' into ForegroundLoad
2026.05.06 08:48:32.837866 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.opentelemetry_span_log' into ForegroundLoad
2026.05.06 08:48:32.837932 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.s3queue_log' into ForegroundLoad
2026.05.06 08:48:32.837998 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.metric_log' into ForegroundLoad
2026.05.06 08:48:32.838065 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.query_log' into ForegroundLoad
2026.05.06 08:48:32.838130 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.text_log' into ForegroundLoad
2026.05.06 08:48:32.838196 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.crash_log' into ForegroundLoad
2026.05.06 08:48:32.838337 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.asynchronous_metric_log' into ForegroundLoad
2026.05.06 08:48:32.838404 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.backup_log' into ForegroundLoad
2026.05.06 08:48:32.838472 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.transactions_info_log' into ForegroundLoad
2026.05.06 08:48:32.838552 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.blob_storage_log' into ForegroundLoad
2026.05.06 08:48:32.838616 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.error_log' into ForegroundLoad
2026.05.06 08:48:32.838681 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.asynchronous_insert_log' into ForegroundLoad
2026.05.06 08:48:32.838747 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.zookeeper_log' into ForegroundLoad
2026.05.06 08:48:32.838813 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table system.processors_profile_log' into ForegroundLoad
2026.05.06 08:48:32.838915 [ 31633 ] {} <Debug> AsyncLoader: Change current priority: none -> 0
2026.05.06 08:48:32.839084 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #1 in ForegroundLoad
2026.05.06 08:48:32.839318 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #2 in ForegroundLoad
2026.05.06 08:48:32.839473 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #3 in ForegroundLoad
2026.05.06 08:48:32.839592 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #4 in ForegroundLoad
2026.05.06 08:48:32.839721 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #5 in ForegroundLoad
2026.05.06 08:48:32.839819 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #6 in ForegroundLoad
2026.05.06 08:48:32.839917 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #7 in ForegroundLoad
2026.05.06 08:48:32.840013 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #8 in ForegroundLoad
2026.05.06 08:48:32.840106 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #9 in ForegroundLoad
2026.05.06 08:48:32.840201 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #10 in ForegroundLoad
2026.05.06 08:48:32.840297 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #11 in ForegroundLoad
2026.05.06 08:48:32.840416 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #12 in ForegroundLoad
2026.05.06 08:48:32.840528 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #13 in ForegroundLoad
2026.05.06 08:48:32.840627 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #14 in ForegroundLoad
2026.05.06 08:48:32.840738 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #15 in ForegroundLoad
2026.05.06 08:48:32.840833 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #16 in ForegroundLoad
2026.05.06 08:48:32.841056 [ 31633 ] {} <Debug> AsyncLoader: Wait load job 'load table system.trace_log' in ForegroundLoad
2026.05.06 08:48:32.841197 [ 32361 ] {} <Debug> AsyncLoader: Execute load job 'load table system.part_log' in ForegroundLoad
2026.05.06 08:48:32.841422 [ 32361 ] {} <Trace> DatabaseAtomic (system): Loading table system.part_log
2026.05.06 08:48:32.841504 [ 32351 ] {} <Debug> AsyncLoader: Execute load job 'load table system.session_log' in ForegroundLoad
2026.05.06 08:48:32.841728 [ 32359 ] {} <Debug> AsyncLoader: Execute load job 'load table system.filesystem_cache_log' in ForegroundLoad
2026.05.06 08:48:32.841723 [ 32351 ] {} <Trace> DatabaseAtomic (system): Loading table system.session_log
2026.05.06 08:48:32.841962 [ 32365 ] {} <Debug> AsyncLoader: Execute load job 'load table system.trace_log' in ForegroundLoad
2026.05.06 08:48:32.841963 [ 32359 ] {} <Trace> DatabaseAtomic (system): Loading table system.filesystem_cache_log
2026.05.06 08:48:32.842122 [ 32365 ] {} <Trace> DatabaseAtomic (system): Loading table system.trace_log
2026.05.06 08:48:32.842136 [ 32349 ] {} <Debug> AsyncLoader: Execute load job 'load table system.query_views_log' in ForegroundLoad
2026.05.06 08:48:32.842285 [ 32349 ] {} <Trace> DatabaseAtomic (system): Loading table system.query_views_log
2026.05.06 08:48:32.842294 [ 32347 ] {} <Debug> AsyncLoader: Execute load job 'load table system.query_thread_log' in ForegroundLoad
2026.05.06 08:48:32.842504 [ 32353 ] {} <Debug> AsyncLoader: Execute load job 'load table system.opentelemetry_span_log' in ForegroundLoad
2026.05.06 08:48:32.842495 [ 32347 ] {} <Trace> DatabaseAtomic (system): Loading table system.query_thread_log
2026.05.06 08:48:32.842658 [ 32353 ] {} <Trace> DatabaseAtomic (system): Loading table system.opentelemetry_span_log
2026.05.06 08:48:32.842662 [ 32360 ] {} <Debug> AsyncLoader: Execute load job 'load table system.s3queue_log' in ForegroundLoad
2026.05.06 08:48:32.842866 [ 32362 ] {} <Debug> AsyncLoader: Execute load job 'load table system.metric_log' in ForegroundLoad
2026.05.06 08:48:32.842843 [ 32360 ] {} <Trace> DatabaseAtomic (system): Loading table system.s3queue_log
2026.05.06 08:48:32.843000 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table system.query_log' in ForegroundLoad
2026.05.06 08:48:32.843051 [ 32339 ] {} <Trace> DatabaseAtomic (system): Loading table system.query_log
2026.05.06 08:48:32.843043 [ 32362 ] {} <Trace> DatabaseAtomic (system): Loading table system.metric_log
2026.05.06 08:48:32.843112 [ 32337 ] {} <Debug> AsyncLoader: Execute load job 'load table system.text_log' in ForegroundLoad
2026.05.06 08:48:32.843229 [ 32337 ] {} <Trace> DatabaseAtomic (system): Loading table system.text_log
2026.05.06 08:48:32.843291 [ 32341 ] {} <Debug> AsyncLoader: Execute load job 'load table system.crash_log' in ForegroundLoad
2026.05.06 08:48:32.843460 [ 32341 ] {} <Trace> DatabaseAtomic (system): Loading table system.crash_log
2026.05.06 08:48:32.843496 [ 32336 ] {} <Debug> AsyncLoader: Execute load job 'load table system.asynchronous_metric_log' in ForegroundLoad
2026.05.06 08:48:32.843633 [ 32336 ] {} <Trace> DatabaseAtomic (system): Loading table system.asynchronous_metric_log
2026.05.06 08:48:32.843662 [ 32344 ] {} <Debug> AsyncLoader: Execute load job 'load table system.backup_log' in ForegroundLoad
2026.05.06 08:48:32.843759 [ 32344 ] {} <Trace> DatabaseAtomic (system): Loading table system.backup_log
2026.05.06 08:48:32.843861 [ 32338 ] {} <Debug> AsyncLoader: Execute load job 'load table system.transactions_info_log' in ForegroundLoad
2026.05.06 08:48:32.843974 [ 32338 ] {} <Trace> DatabaseAtomic (system): Loading table system.transactions_info_log
2026.05.06 08:48:32.844015 [ 32343 ] {} <Debug> AsyncLoader: Execute load job 'load table system.blob_storage_log' in ForegroundLoad
2026.05.06 08:48:32.844096 [ 32343 ] {} <Trace> DatabaseAtomic (system): Loading table system.blob_storage_log
2026.05.06 08:48:32.869649 [ 32353 ] {} <Debug> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading data parts
2026.05.06 08:48:32.873019 [ 32353 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.873183 [ 32353 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Will load 2 parts using up to 64 threads
2026.05.06 08:48:32.873593 [ 32342 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Active part 202605_1_153_41 from disk default
2026.05.06 08:48:32.873724 [ 32340 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Active part 202605_154_154_0 from disk default
2026.05.06 08:48:32.887516 [ 32342 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_153_41 (state Active) into data_parts_indexes
2026.05.06 08:48:32.887703 [ 32342 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Active part 202605_1_153_41 on disk default
2026.05.06 08:48:32.887730 [ 32340 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_154_154_0 (state Active) into data_parts_indexes
2026.05.06 08:48:32.887846 [ 32340 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Active part 202605_154_154_0 on disk default
2026.05.06 08:48:32.888244 [ 32353 ] {} <Debug> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Found 57 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:32.888405 [ 32353 ] {} <Debug> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loaded data parts (2 items)
2026.05.06 08:48:32.890801 [ 32353 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:32.890931 [ 32353 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:32.891095 [ 32353 ] {} <Debug> AsyncLoader: Finish load job 'load table system.opentelemetry_span_log' with status OK
2026.05.06 08:48:32.891283 [ 32353 ] {} <Debug> AsyncLoader: Execute load job 'load table system.error_log' in ForegroundLoad
2026.05.06 08:48:32.891359 [ 32353 ] {} <Trace> DatabaseAtomic (system): Loading table system.error_log
2026.05.06 08:48:32.915738 [ 32341 ] {} <Debug> system.crash_log (5ca3b46d-763a-47f5-8e69-37d0db59a8ce): Loading data parts
2026.05.06 08:48:32.916574 [ 32341 ] {} <Test> system.crash_log (5ca3b46d-763a-47f5-8e69-37d0db59a8ce): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.916667 [ 32341 ] {} <Trace> system.crash_log (5ca3b46d-763a-47f5-8e69-37d0db59a8ce): Will load 1 parts using up to 64 threads
2026.05.06 08:48:32.917302 [ 32365 ] {} <Debug> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading data parts
2026.05.06 08:48:32.917580 [ 32340 ] {} <Trace> system.crash_log (5ca3b46d-763a-47f5-8e69-37d0db59a8ce): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:32.918339 [ 32344 ] {} <Debug> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): Loading data parts
2026.05.06 08:48:32.919308 [ 32337 ] {} <Debug> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading data parts
2026.05.06 08:48:32.919517 [ 32359 ] {} <Debug> system.filesystem_cache_log (0d3814ca-4f0b-456d-98ab-09a90c6aee70): Loading data parts
2026.05.06 08:48:32.920623 [ 32353 ] {} <Debug> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading data parts
2026.05.06 08:48:32.921561 [ 32360 ] {} <Debug> system.s3queue_log (c09fbdf4-3a59-4472-8d00-54dd271a95b2): Loading data parts
2026.05.06 08:48:32.921624 [ 32339 ] {} <Debug> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading data parts
2026.05.06 08:48:32.922532 [ 32351 ] {} <Debug> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loading data parts
2026.05.06 08:48:32.923274 [ 32360 ] {} <Test> system.s3queue_log (c09fbdf4-3a59-4472-8d00-54dd271a95b2): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.923311 [ 32359 ] {} <Test> system.filesystem_cache_log (0d3814ca-4f0b-456d-98ab-09a90c6aee70): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.923637 [ 32336 ] {} <Debug> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading data parts
2026.05.06 08:48:32.923669 [ 32359 ] {} <Debug> system.filesystem_cache_log (0d3814ca-4f0b-456d-98ab-09a90c6aee70): There are no data parts
2026.05.06 08:48:32.923669 [ 32360 ] {} <Debug> system.s3queue_log (c09fbdf4-3a59-4472-8d00-54dd271a95b2): There are no data parts
2026.05.06 08:48:32.923733 [ 32349 ] {} <Debug> system.query_views_log (1eb2444b-07f6-41ea-a56f-9e2fd3310474): Loading data parts
2026.05.06 08:48:32.923757 [ 32351 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.923859 [ 32340 ] {} <Test> system.crash_log (5ca3b46d-763a-47f5-8e69-37d0db59a8ce): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:32.923910 [ 32340 ] {} <Trace> system.crash_log (5ca3b46d-763a-47f5-8e69-37d0db59a8ce): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:32.923942 [ 32351 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Will load 3 parts using up to 64 threads
2026.05.06 08:48:32.924024 [ 32343 ] {} <Debug> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading data parts
2026.05.06 08:48:32.924069 [ 32344 ] {} <Test> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.924104 [ 32341 ] {} <Debug> system.crash_log (5ca3b46d-763a-47f5-8e69-37d0db59a8ce): Loaded data parts (1 items)
2026.05.06 08:48:32.924122 [ 32344 ] {} <Trace> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): Will load 3 parts using up to 64 threads
2026.05.06 08:48:32.924546 [ 32349 ] {} <Test> system.query_views_log (1eb2444b-07f6-41ea-a56f-9e2fd3310474): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.924633 [ 32349 ] {} <Trace> system.query_views_log (1eb2444b-07f6-41ea-a56f-9e2fd3310474): Will load 2 parts using up to 64 threads
2026.05.06 08:48:32.924623 [ 32338 ] {} <Debug> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading data parts
2026.05.06 08:48:32.925021 [ 32360 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:32.925139 [ 32360 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:32.924586 [ 32368 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loading Active part 202605_1_20_4 from disk default
2026.05.06 08:48:32.925291 [ 32360 ] {} <Debug> AsyncLoader: Finish load job 'load table system.s3queue_log' with status OK
2026.05.06 08:48:32.925278 [ 32373 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loading Active part 202605_21_21_0 from disk default
2026.05.06 08:48:32.925397 [ 32365 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.925424 [ 32360 ] {} <Debug> AsyncLoader: Execute load job 'load table system.asynchronous_insert_log' in ForegroundLoad
2026.05.06 08:48:32.925444 [ 32341 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:32.925448 [ 32365 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Will load 6 parts using up to 64 threads
2026.05.06 08:48:32.925729 [ 32360 ] {} <Trace> DatabaseAtomic (system): Loading table system.asynchronous_insert_log
2026.05.06 08:48:32.925821 [ 32361 ] {} <Debug> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading data parts
2026.05.06 08:48:32.925913 [ 32341 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:32.926290 [ 32341 ] {} <Debug> AsyncLoader: Finish load job 'load table system.crash_log' with status OK
2026.05.06 08:48:32.926398 [ 32341 ] {} <Debug> AsyncLoader: Execute load job 'load table system.zookeeper_log' in ForegroundLoad
2026.05.06 08:48:32.926501 [ 32341 ] {} <Trace> DatabaseAtomic (system): Loading table system.zookeeper_log
2026.05.06 08:48:32.926935 [ 32359 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:32.927344 [ 32340 ] {} <Trace> system.query_views_log (1eb2444b-07f6-41ea-a56f-9e2fd3310474): Loading Active part 202605_32_32_0 from disk default
2026.05.06 08:48:32.927131 [ 32367 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Active part 202605_172_172_0 from disk default
2026.05.06 08:48:32.927155 [ 32370 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loading Active part 202605_22_22_0 from disk default
2026.05.06 08:48:32.927573 [ 32339 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.927578 [ 32359 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:32.927714 [ 32339 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Will load 3 parts using up to 64 threads
2026.05.06 08:48:32.928034 [ 32359 ] {} <Debug> AsyncLoader: Finish load job 'load table system.filesystem_cache_log' with status OK
2026.05.06 08:48:32.928151 [ 32359 ] {} <Debug> AsyncLoader: Execute load job 'load table system.processors_profile_log' in ForegroundLoad
2026.05.06 08:48:32.928236 [ 32359 ] {} <Trace> DatabaseAtomic (system): Loading table system.processors_profile_log
2026.05.06 08:48:32.929169 [ 32342 ] {} <Trace> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): Loading Active part 202605_2_2_0 from disk default
2026.05.06 08:48:32.930067 [ 32347 ] {} <Debug> system.query_thread_log (bbd8f07e-2292-4bbc-a894-d08be289f303): Loading data parts
2026.05.06 08:48:32.930078 [ 32343 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.930238 [ 32343 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Will load 1 parts using up to 64 threads
2026.05.06 08:48:32.932824 [ 32353 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.932927 [ 32353 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Will load 3 parts using up to 64 threads
2026.05.06 08:48:32.930143 [ 32369 ] {} <Trace> system.query_views_log (1eb2444b-07f6-41ea-a56f-9e2fd3310474): Loading Active part 202605_1_31_6 from disk default
2026.05.06 08:48:32.934238 [ 32338 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.934346 [ 32338 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Will load 5 parts using up to 64 threads
2026.05.06 08:48:32.932430 [ 32371 ] {} <Trace> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): Loading Active part 202605_3_3_0 from disk default
2026.05.06 08:48:32.933215 [ 32375 ] {} <Trace> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): Loading Active part 202605_1_1_0 from disk default
2026.05.06 08:48:32.932737 [ 32372 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Active part 202605_155_155_0 from disk default
2026.05.06 08:48:32.935842 [ 32336 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.935944 [ 32336 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Will load 5 parts using up to 64 threads
2026.05.06 08:48:32.938060 [ 32374 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Active part 202605_174_174_0 from disk default
2026.05.06 08:48:32.938148 [ 32376 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Active part 202605_1_122_16 from disk default
2026.05.06 08:48:32.939961 [ 32361 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.940051 [ 32361 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Will load 3 parts using up to 64 threads
2026.05.06 08:48:32.942122 [ 32337 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.942228 [ 32337 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Will load 6 parts using up to 64 threads
2026.05.06 08:48:32.940020 [ 32378 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Active part 202605_1_10_2 from disk default
2026.05.06 08:48:32.947558 [ 32347 ] {} <Test> system.query_thread_log (bbd8f07e-2292-4bbc-a894-d08be289f303): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:32.947676 [ 32347 ] {} <Debug> system.query_thread_log (bbd8f07e-2292-4bbc-a894-d08be289f303): There are no data parts
2026.05.06 08:48:32.949235 [ 32347 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:32.949316 [ 32347 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:32.949429 [ 32347 ] {} <Debug> AsyncLoader: Finish load job 'load table system.query_thread_log' with status OK
2026.05.06 08:48:32.947269 [ 32379 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Active part 202605_1_166_33 from disk default
2026.05.06 08:48:32.955836 [ 32347 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:32.958698 [ 32347 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Active part 202605_1_171_34 from disk default
2026.05.06 08:48:32.959223 [ 32380 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Active part 202605_167_167_0 from disk default
2026.05.06 08:48:32.959292 [ 32383 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Active part 202605_168_168_0 from disk default
2026.05.06 08:48:32.956178 [ 32377 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Active part 202605_173_173_0 from disk default
2026.05.06 08:48:32.958055 [ 32381 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Active part 202605_123_154_6 from disk default
2026.05.06 08:48:32.964768 [ 32359 ] {} <Debug> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading data parts
2026.05.06 08:48:32.966102 [ 32342 ] {} <Test> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): loadDataPart: inserting 202605_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:32.966203 [ 32342 ] {} <Trace> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): Finished loading Active part 202605_2_2_0 on disk default
2026.05.06 08:48:32.966138 [ 32384 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Active part 202605_175_175_0 from disk default
2026.05.06 08:48:32.968736 [ 32340 ] {} <Test> system.query_views_log (1eb2444b-07f6-41ea-a56f-9e2fd3310474): loadDataPart: inserting 202605_32_32_0 (state Active) into data_parts_indexes
2026.05.06 08:48:32.969137 [ 32340 ] {} <Trace> system.query_views_log (1eb2444b-07f6-41ea-a56f-9e2fd3310474): Finished loading Active part 202605_32_32_0 on disk default
2026.05.06 08:48:32.966138 [ 32386 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Active part 202605_183_183_0 from disk default
2026.05.06 08:48:32.971229 [ 32342 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Active part 202605_1_146_29 from disk default
2026.05.06 08:48:32.973621 [ 32382 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Active part 202605_148_148_0 from disk default
2026.05.06 08:48:32.973617 [ 32385 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Active part 202605_173_173_0 from disk default
2026.05.06 08:48:32.974994 [ 32367 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_172_172_0 (state Active) into data_parts_indexes
2026.05.06 08:48:32.975099 [ 32367 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Active part 202605_172_172_0 on disk default
2026.05.06 08:48:32.972780 [ 32387 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Active part 202605_1_170_75 from disk default
2026.05.06 08:48:32.974843 [ 32388 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Active part 202605_310_310_0 from disk default
2026.05.06 08:48:32.981959 [ 32340 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Active part 202605_301_306_1 from disk default
2026.05.06 08:48:32.973773 [ 32390 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Active part 202605_1_181_36 from disk default
2026.05.06 08:48:32.986936 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Active part 202605_307_307_0 from disk default
2026.05.06 08:48:32.987216 [ 32367 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Active part 202605_1_300_16 from disk default
2026.05.06 08:48:32.989230 [ 32380 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_167_167_0 (state Active) into data_parts_indexes
2026.05.06 08:48:32.989343 [ 32380 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Active part 202605_167_167_0 on disk default
2026.05.06 08:48:32.986997 [ 32391 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Active part 202605_171_171_0 from disk default
2026.05.06 08:48:32.992096 [ 32393 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Active part 202605_175_175_0 from disk default
2026.05.06 08:48:32.993974 [ 32392 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Active part 202605_309_309_0 from disk default
2026.05.06 08:48:33.007634 [ 32380 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Active part 202605_184_184_0 from disk default
2026.05.06 08:48:33.008940 [ 32371 ] {} <Test> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): loadDataPart: inserting 202605_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.009050 [ 32371 ] {} <Trace> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): Finished loading Active part 202605_3_3_0 on disk default
2026.05.06 08:48:33.009271 [ 32371 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Active part 202605_147_147_0 from disk default
2026.05.06 08:48:33.013316 [ 32340 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_301_306_1 (state Active) into data_parts_indexes
2026.05.06 08:48:33.013410 [ 32340 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Active part 202605_301_306_1 on disk default
2026.05.06 08:48:33.015778 [ 32369 ] {} <Test> system.query_views_log (1eb2444b-07f6-41ea-a56f-9e2fd3310474): loadDataPart: inserting 202605_1_31_6 (state Active) into data_parts_indexes
2026.05.06 08:48:33.015877 [ 32369 ] {} <Trace> system.query_views_log (1eb2444b-07f6-41ea-a56f-9e2fd3310474): Finished loading Active part 202605_1_31_6 on disk default
2026.05.06 08:48:32.997073 [ 32396 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Active part 202605_185_185_0 from disk default
2026.05.06 08:48:33.019142 [ 32349 ] {} <Debug> system.query_views_log (1eb2444b-07f6-41ea-a56f-9e2fd3310474): Loaded data parts (2 items)
2026.05.06 08:48:33.017215 [ 32397 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Active part 202605_172_172_0 from disk default
2026.05.06 08:48:33.021445 [ 32349 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.021544 [ 32349 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.021666 [ 32349 ] {} <Debug> AsyncLoader: Finish load job 'load table system.query_views_log' with status OK
2026.05.06 08:48:33.021752 [ 32349 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.022616 [ 32360 ] {} <Debug> system.asynchronous_insert_log (7d0815d4-fc06-4b2a-92c9-04808d624a7c): Loading data parts
2026.05.06 08:48:33.026558 [ 32359 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:33.026671 [ 32359 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Will load 5 parts using up to 64 threads
2026.05.06 08:48:33.027125 [ 32360 ] {} <Test> system.asynchronous_insert_log (7d0815d4-fc06-4b2a-92c9-04808d624a7c): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:33.027222 [ 32360 ] {} <Trace> system.asynchronous_insert_log (7d0815d4-fc06-4b2a-92c9-04808d624a7c): Will load 1 parts using up to 64 threads
2026.05.06 08:48:33.028294 [ 32378 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_1_10_2 (state Active) into data_parts_indexes
2026.05.06 08:48:33.028392 [ 32378 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Active part 202605_1_10_2 on disk default
2026.05.06 08:48:33.030628 [ 32380 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_184_184_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.030733 [ 32380 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Active part 202605_184_184_0 on disk default
2026.05.06 08:48:33.020032 [ 32399 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Active part 202605_182_182_0 from disk default
2026.05.06 08:48:33.019971 [ 32398 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Active part 202605_308_308_0 from disk default
2026.05.06 08:48:33.023825 [ 32395 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Active part 202605_174_174_0 from disk default
2026.05.06 08:48:33.033880 [ 32378 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Active part 202605_152_152_0 from disk default
2026.05.06 08:48:33.033964 [ 32349 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Active part 202605_1_150_48 from disk default
2026.05.06 08:48:33.034022 [ 32340 ] {} <Trace> system.asynchronous_insert_log (7d0815d4-fc06-4b2a-92c9-04808d624a7c): Loading Active part 20260506_1_1_0 from disk default
2026.05.06 08:48:33.035049 [ 32343 ] {} <Debug> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Found 11 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.035171 [ 32343 ] {} <Debug> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loaded data parts (1 items)
2026.05.06 08:48:33.037065 [ 32374 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_174_174_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.037268 [ 32374 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Active part 202605_174_174_0 on disk default
2026.05.06 08:48:33.043633 [ 32379 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_1_166_33 (state Active) into data_parts_indexes
2026.05.06 08:48:33.043644 [ 32375 ] {} <Test> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): loadDataPart: inserting 202605_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.043755 [ 32379 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Active part 202605_1_166_33 on disk default
2026.05.06 08:48:33.043761 [ 32386 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_183_183_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.043814 [ 32343 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.043813 [ 32375 ] {} <Trace> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): Finished loading Active part 202605_1_1_0 on disk default
2026.05.06 08:48:33.043875 [ 32386 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Active part 202605_183_183_0 on disk default
2026.05.06 08:48:33.043909 [ 32343 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.044011 [ 32343 ] {} <Debug> AsyncLoader: Finish load job 'load table system.blob_storage_log' with status OK
2026.05.06 08:48:33.044077 [ 32343 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.044426 [ 32377 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_173_173_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.044519 [ 32377 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Active part 202605_173_173_0 on disk default
2026.05.06 08:48:33.044537 [ 32344 ] {} <Debug> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): Loaded data parts (3 items)
2026.05.06 08:48:33.046855 [ 32344 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.051930 [ 32344 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.051958 [ 32400 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Active part 202605_154_154_0 from disk default
2026.05.06 08:48:33.052070 [ 32344 ] {} <Debug> AsyncLoader: Finish load job 'load table system.backup_log' with status OK
2026.05.06 08:48:33.052198 [ 32344 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.052818 [ 32383 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_168_168_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.052918 [ 32383 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Active part 202605_168_168_0 on disk default
2026.05.06 08:48:33.053517 [ 32341 ] {} <Debug> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading data parts
2026.05.06 08:48:33.055095 [ 32388 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_310_310_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.055194 [ 32388 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Active part 202605_310_310_0 on disk default
2026.05.06 08:48:33.055561 [ 32353 ] {} <Debug> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Found 54 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.055694 [ 32353 ] {} <Debug> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loaded data parts (3 items)
2026.05.06 08:48:33.057424 [ 32378 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_152_152_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.057541 [ 32378 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Active part 202605_152_152_0 on disk default
2026.05.06 08:48:33.054094 [ 32402 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Active part 202605_151_151_0 from disk default
2026.05.06 08:48:33.062496 [ 32353 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.062606 [ 32353 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.062757 [ 32353 ] {} <Debug> AsyncLoader: Finish load job 'load table system.error_log' with status OK
2026.05.06 08:48:33.062844 [ 32353 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.063066 [ 32399 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_182_182_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.063168 [ 32399 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Active part 202605_182_182_0 on disk default
2026.05.06 08:48:33.067282 [ 32394 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_307_307_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.067398 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Active part 202605_307_307_0 on disk default
2026.05.06 08:48:33.058943 [ 32401 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Active part 202605_153_153_0 from disk default
2026.05.06 08:48:33.071064 [ 32371 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_147_147_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.071159 [ 32371 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Active part 202605_147_147_0 on disk default
2026.05.06 08:48:33.072329 [ 32393 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_175_175_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.072384 [ 32391 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_171_171_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.072429 [ 32393 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Active part 202605_175_175_0 on disk default
2026.05.06 08:48:33.072499 [ 32391 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Active part 202605_171_171_0 on disk default
2026.05.06 08:48:33.074535 [ 32384 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_175_175_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.072664 [ 32342 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_1_146_29 (state Active) into data_parts_indexes
2026.05.06 08:48:33.074643 [ 32384 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Active part 202605_175_175_0 on disk default
2026.05.06 08:48:33.074736 [ 32342 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Active part 202605_1_146_29 on disk default
2026.05.06 08:48:33.076709 [ 32382 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_148_148_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.076823 [ 32382 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Active part 202605_148_148_0 on disk default
2026.05.06 08:48:33.076686 [ 32341 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:33.077045 [ 32341 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Will load 5 parts using up to 64 threads
2026.05.06 08:48:33.077054 [ 32385 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_173_173_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.077184 [ 32385 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Active part 202605_173_173_0 on disk default
2026.05.06 08:48:33.077316 [ 32380 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Active part 202605_168_168_0 from disk default
2026.05.06 08:48:33.077382 [ 32385 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Active part 202605_169_169_0 from disk default
2026.05.06 08:48:33.077501 [ 32374 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Active part 202605_167_167_0 from disk default
2026.05.06 08:48:33.077969 [ 32379 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Active part 202605_170_170_0 from disk default
2026.05.06 08:48:33.078429 [ 32347 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_1_171_34 (state Active) into data_parts_indexes
2026.05.06 08:48:33.078536 [ 32347 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Active part 202605_1_171_34 on disk default
2026.05.06 08:48:33.079870 [ 32361 ] {} <Debug> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Found 48 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.079919 [ 32404 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Active part 202605_1_166_52 from disk default
2026.05.06 08:48:33.080093 [ 32361 ] {} <Debug> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loaded data parts (3 items)
2026.05.06 08:48:33.079913 [ 32390 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_1_181_36 (state Active) into data_parts_indexes
2026.05.06 08:48:33.080380 [ 32390 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Active part 202605_1_181_36 on disk default
2026.05.06 08:48:33.083373 [ 32396 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_185_185_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.083444 [ 32361 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.083469 [ 32396 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Active part 202605_185_185_0 on disk default
2026.05.06 08:48:33.083593 [ 32361 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.083778 [ 32336 ] {} <Debug> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Found 54 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.083787 [ 32361 ] {} <Debug> AsyncLoader: Finish load job 'load table system.part_log' with status OK
2026.05.06 08:48:33.084121 [ 32336 ] {} <Debug> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loaded data parts (5 items)
2026.05.06 08:48:33.084335 [ 32361 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.086954 [ 32336 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.087051 [ 32398 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_308_308_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.087047 [ 32336 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.087107 [ 32398 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Active part 202605_308_308_0 on disk default
2026.05.06 08:48:33.087201 [ 32336 ] {} <Debug> AsyncLoader: Finish load job 'load table system.asynchronous_metric_log' with status OK
2026.05.06 08:48:33.087309 [ 32336 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.087329 [ 32368 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): loadDataPart: inserting 202605_1_20_4 (state Active) into data_parts_indexes
2026.05.06 08:48:33.087533 [ 32368 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Finished loading Active part 202605_1_20_4 on disk default
2026.05.06 08:48:33.087574 [ 32370 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): loadDataPart: inserting 202605_22_22_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.087689 [ 32373 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): loadDataPart: inserting 202605_21_21_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.087707 [ 32370 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Finished loading Active part 202605_22_22_0 on disk default
2026.05.06 08:48:33.087730 [ 32373 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Finished loading Active part 202605_21_21_0 on disk default
2026.05.06 08:48:33.088727 [ 32397 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_172_172_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.088810 [ 32397 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Active part 202605_172_172_0 on disk default
2026.05.06 08:48:33.089009 [ 32340 ] {} <Test> system.asynchronous_insert_log (7d0815d4-fc06-4b2a-92c9-04808d624a7c): loadDataPart: inserting 20260506_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.089100 [ 32340 ] {} <Trace> system.asynchronous_insert_log (7d0815d4-fc06-4b2a-92c9-04808d624a7c): Finished loading Active part 20260506_1_1_0 on disk default
2026.05.06 08:48:33.089278 [ 32392 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_309_309_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.089318 [ 32392 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Active part 202605_309_309_0 on disk default
2026.05.06 08:48:33.089460 [ 32400 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_154_154_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.089507 [ 32400 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Active part 202605_154_154_0 on disk default
2026.05.06 08:48:33.090742 [ 32351 ] {} <Debug> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Found 6 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.090788 [ 32402 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_151_151_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.090869 [ 32402 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Active part 202605_151_151_0 on disk default
2026.05.06 08:48:33.090863 [ 32351 ] {} <Debug> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loaded data parts (3 items)
2026.05.06 08:48:33.090720 [ 32360 ] {} <Debug> system.asynchronous_insert_log (7d0815d4-fc06-4b2a-92c9-04808d624a7c): Loaded data parts (1 items)
2026.05.06 08:48:33.091146 [ 32349 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_150_48 (state Active) into data_parts_indexes
2026.05.06 08:48:33.091238 [ 32349 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Active part 202605_1_150_48 on disk default
2026.05.06 08:48:33.092060 [ 32387 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_170_75 (state Active) into data_parts_indexes
2026.05.06 08:48:33.092172 [ 32387 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Active part 202605_1_170_75 on disk default
2026.05.06 08:48:33.092232 [ 32351 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.092315 [ 32351 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.092459 [ 32351 ] {} <Debug> AsyncLoader: Finish load job 'load table system.session_log' with status OK
2026.05.06 08:48:33.092455 [ 32360 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.092561 [ 32351 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.092573 [ 32360 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.092709 [ 32360 ] {} <Debug> AsyncLoader: Finish load job 'load table system.asynchronous_insert_log' with status OK
2026.05.06 08:48:33.092705 [ 32365 ] {} <Debug> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Found 61 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.092809 [ 32360 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.092851 [ 32365 ] {} <Debug> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loaded data parts (6 items)
2026.05.06 08:48:33.093178 [ 32367 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_1_300_16 (state Active) into data_parts_indexes
2026.05.06 08:48:33.093275 [ 32367 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Active part 202605_1_300_16 on disk default
2026.05.06 08:48:33.093768 [ 32337 ] {} <Debug> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Found 91 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.093857 [ 32337 ] {} <Debug> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loaded data parts (6 items)
2026.05.06 08:48:33.094784 [ 32401 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_153_153_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.094878 [ 32401 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Active part 202605_153_153_0 on disk default
2026.05.06 08:48:33.095252 [ 32365 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.095315 [ 32359 ] {} <Debug> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Found 50 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.095345 [ 32365 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.095419 [ 32359 ] {} <Debug> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loaded data parts (5 items)
2026.05.06 08:48:33.095474 [ 32365 ] {} <Debug> AsyncLoader: Finish load job 'load table system.trace_log' with status OK
2026.05.06 08:48:33.095859 [ 32365 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.096031 [ 31633 ] {} <Debug> AsyncLoader: Wait load job 'load table system.transactions_info_log' in ForegroundLoad
2026.05.06 08:48:33.096395 [ 32337 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.096507 [ 32337 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.096629 [ 32337 ] {} <Debug> AsyncLoader: Finish load job 'load table system.text_log' with status OK
2026.05.06 08:48:33.096716 [ 32337 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.096880 [ 32372 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_155_155_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.096967 [ 32372 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Active part 202605_155_155_0 on disk default
2026.05.06 08:48:33.097277 [ 32359 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.097368 [ 32359 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.097618 [ 32395 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_174_174_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.097637 [ 32359 ] {} <Debug> AsyncLoader: Finish load job 'load table system.processors_profile_log' with status OK
2026.05.06 08:48:33.097719 [ 32395 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Active part 202605_174_174_0 on disk default
2026.05.06 08:48:33.097794 [ 32359 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.098070 [ 32338 ] {} <Debug> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Found 48 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.098160 [ 32338 ] {} <Debug> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loaded data parts (5 items)
2026.05.06 08:48:33.099826 [ 32338 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.099906 [ 32338 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.100036 [ 32338 ] {} <Debug> AsyncLoader: Finish load job 'load table system.transactions_info_log' with status OK
2026.05.06 08:48:33.100119 [ 32338 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.100328 [ 31633 ] {} <Debug> AsyncLoader: Wait load job 'load table system.zookeeper_log' in ForegroundLoad
2026.05.06 08:48:33.101008 [ 32380 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_168_168_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.101066 [ 32380 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Active part 202605_168_168_0 on disk default
2026.05.06 08:48:33.103411 [ 32385 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_169_169_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.103451 [ 32385 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Active part 202605_169_169_0 on disk default
2026.05.06 08:48:33.104181 [ 32404 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_166_52 (state Active) into data_parts_indexes
2026.05.06 08:48:33.104227 [ 32404 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Active part 202605_1_166_52 on disk default
2026.05.06 08:48:33.104336 [ 32374 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_167_167_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.104433 [ 32374 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Active part 202605_167_167_0 on disk default
2026.05.06 08:48:33.106221 [ 32381 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_123_154_6 (state Active) into data_parts_indexes
2026.05.06 08:48:33.106295 [ 32381 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Active part 202605_123_154_6 on disk default
2026.05.06 08:48:33.106905 [ 32379 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_170_170_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.106983 [ 32379 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Active part 202605_170_170_0 on disk default
2026.05.06 08:48:33.107468 [ 32341 ] {} <Debug> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Found 57 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.107684 [ 32341 ] {} <Debug> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loaded data parts (5 items)
2026.05.06 08:48:33.109982 [ 32341 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.110075 [ 32341 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.110248 [ 32341 ] {} <Debug> AsyncLoader: Finish load job 'load table system.zookeeper_log' with status OK
2026.05.06 08:48:33.110354 [ 32341 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.110497 [ 31633 ] {} <Debug> AsyncLoader: Wait load job 'load table system.query_log' in ForegroundLoad
2026.05.06 08:48:33.124496 [ 32376 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_1_122_16 (state Active) into data_parts_indexes
2026.05.06 08:48:33.124559 [ 32376 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Active part 202605_1_122_16 on disk default
2026.05.06 08:48:33.124923 [ 32339 ] {} <Debug> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Found 48 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.124988 [ 32339 ] {} <Debug> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loaded data parts (3 items)
2026.05.06 08:48:33.126201 [ 32339 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.126236 [ 32339 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.126293 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table system.query_log' with status OK
2026.05.06 08:48:33.126330 [ 32339 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.126454 [ 31633 ] {} <Debug> AsyncLoader: Wait load job 'load table system.metric_log' in ForegroundLoad
2026.05.06 08:48:33.219155 [ 32362 ] {} <Debug> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading data parts
2026.05.06 08:48:33.221012 [ 32362 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:33.221059 [ 32362 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Will load 4 parts using up to 64 threads
2026.05.06 08:48:33.221231 [ 32386 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Active part 202605_1_131_26 from disk default
2026.05.06 08:48:33.221281 [ 32343 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Active part 202605_132_132_0 from disk default
2026.05.06 08:48:33.221406 [ 32377 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Active part 202605_134_134_0 from disk default
2026.05.06 08:48:33.221514 [ 32344 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Active part 202605_133_133_0 from disk default
2026.05.06 08:48:33.350981 [ 32386 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_1_131_26 (state Active) into data_parts_indexes
2026.05.06 08:48:33.351052 [ 32386 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Active part 202605_1_131_26 on disk default
2026.05.06 08:48:33.356106 [ 32343 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_132_132_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.356166 [ 32343 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Active part 202605_132_132_0 on disk default
2026.05.06 08:48:33.356173 [ 32377 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_134_134_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.356227 [ 32377 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Active part 202605_134_134_0 on disk default
2026.05.06 08:48:33.359330 [ 32344 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_133_133_0 (state Active) into data_parts_indexes
2026.05.06 08:48:33.359407 [ 32344 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Active part 202605_133_133_0 on disk default
2026.05.06 08:48:33.363853 [ 32362 ] {} <Debug> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Found 36 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:33.363972 [ 32362 ] {} <Debug> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loaded data parts (4 items)
2026.05.06 08:48:33.374412 [ 32362 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:33.374512 [ 32362 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:33.374925 [ 32362 ] {} <Debug> AsyncLoader: Finish load job 'load table system.metric_log' with status OK
2026.05.06 08:48:33.375013 [ 32362 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.375118 [ 32362 ] {} <Debug> AsyncLoader: Change current priority: 0 -> none
2026.05.06 08:48:33.412468 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.trace_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.412650 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.transactions_info_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.412760 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.s3queue_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.412835 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.query_thread_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.412908 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.session_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.412979 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.part_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413051 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.backup_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413123 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.asynchronous_metric_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413195 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.error_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413266 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.filesystem_cache_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413338 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.text_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413409 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.crash_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413511 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.opentelemetry_span_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413582 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.blob_storage_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413654 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.zookeeper_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413723 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.query_views_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413792 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.processors_profile_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413863 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.query_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.413933 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.metric_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.414020 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table system.asynchronous_insert_log': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.414091 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup Atomic database system': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.414229 [ 31633 ] {} <Debug> AsyncLoader: Prioritize load job 'startup Ordinary database system': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:33.414600 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.backup_log' into ForegroundLoad
2026.05.06 08:48:33.414719 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.s3queue_log' into ForegroundLoad
2026.05.06 08:48:33.414786 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.metric_log' into ForegroundLoad
2026.05.06 08:48:33.414850 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.query_views_log' into ForegroundLoad
2026.05.06 08:48:33.414913 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.error_log' into ForegroundLoad
2026.05.06 08:48:33.414976 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database system' into ForegroundLoad
2026.05.06 08:48:33.415038 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.session_log' into ForegroundLoad
2026.05.06 08:48:33.415102 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.asynchronous_insert_log' into ForegroundLoad
2026.05.06 08:48:33.415167 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.transactions_info_log' into ForegroundLoad
2026.05.06 08:48:33.415243 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.trace_log' into ForegroundLoad
2026.05.06 08:48:33.415314 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.part_log' into ForegroundLoad
2026.05.06 08:48:33.415378 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.processors_profile_log' into ForegroundLoad
2026.05.06 08:48:33.415443 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.asynchronous_metric_log' into ForegroundLoad
2026.05.06 08:48:33.415525 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.filesystem_cache_log' into ForegroundLoad
2026.05.06 08:48:33.415588 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.query_thread_log' into ForegroundLoad
2026.05.06 08:48:33.415679 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.zookeeper_log' into ForegroundLoad
2026.05.06 08:48:33.415742 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.text_log' into ForegroundLoad
2026.05.06 08:48:33.415805 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database system' into ForegroundLoad
2026.05.06 08:48:33.415876 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.crash_log' into ForegroundLoad
2026.05.06 08:48:33.415978 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.opentelemetry_span_log' into ForegroundLoad
2026.05.06 08:48:33.416066 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.blob_storage_log' into ForegroundLoad
2026.05.06 08:48:33.416129 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table system.query_log' into ForegroundLoad
2026.05.06 08:48:33.416224 [ 31633 ] {} <Debug> AsyncLoader: Change current priority: none -> 0
2026.05.06 08:48:33.416358 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #1 in ForegroundLoad
2026.05.06 08:48:33.416539 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #2 in ForegroundLoad
2026.05.06 08:48:33.416625 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #3 in ForegroundLoad
2026.05.06 08:48:33.416707 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #4 in ForegroundLoad
2026.05.06 08:48:33.416814 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #5 in ForegroundLoad
2026.05.06 08:48:33.416986 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #6 in ForegroundLoad
2026.05.06 08:48:33.417102 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #7 in ForegroundLoad
2026.05.06 08:48:33.417184 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #8 in ForegroundLoad
2026.05.06 08:48:33.417262 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #9 in ForegroundLoad
2026.05.06 08:48:33.417341 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #10 in ForegroundLoad
2026.05.06 08:48:33.417419 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #11 in ForegroundLoad
2026.05.06 08:48:33.417522 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #12 in ForegroundLoad
2026.05.06 08:48:33.417599 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #13 in ForegroundLoad
2026.05.06 08:48:33.417680 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #14 in ForegroundLoad
2026.05.06 08:48:33.417802 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #15 in ForegroundLoad
2026.05.06 08:48:33.417885 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #16 in ForegroundLoad
2026.05.06 08:48:33.418093 [ 31633 ] {} <Debug> AsyncLoader: Wait load job 'startup table system.trace_log' in ForegroundLoad
2026.05.06 08:48:33.418109 [ 32383 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.backup_log' in ForegroundLoad
2026.05.06 08:48:33.418280 [ 32378 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.s3queue_log' in ForegroundLoad
2026.05.06 08:48:33.418471 [ 32353 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.metric_log' in ForegroundLoad
2026.05.06 08:48:33.418662 [ 32399 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.query_views_log' in ForegroundLoad
2026.05.06 08:48:33.418859 [ 32369 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.error_log' in ForegroundLoad
2026.05.06 08:48:33.418997 [ 32378 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.s3queue_log' with status OK
2026.05.06 08:48:33.419107 [ 32378 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.session_log' in ForegroundLoad
2026.05.06 08:48:33.419254 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.asynchronous_insert_log' in ForegroundLoad
2026.05.06 08:48:33.419286 [ 31686 ] {} <Debug> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee) (MergerMutator): Selected 3 parts from 202605_1_1_0 to 202605_3_3_0
2026.05.06 08:48:33.419410 [ 32393 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.transactions_info_log' in ForegroundLoad
2026.05.06 08:48:33.419599 [ 32384 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.trace_log' in ForegroundLoad
2026.05.06 08:48:33.419581 [ 31687 ] {} <Debug> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading 36 outdated data parts asynchronously
2026.05.06 08:48:33.419812 [ 32391 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.part_log' in ForegroundLoad
2026.05.06 08:48:33.419797 [ 31690 ] {} <Debug> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loading 6 outdated data parts asynchronously
2026.05.06 08:48:33.419944 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.asynchronous_insert_log' with status OK
2026.05.06 08:48:33.419979 [ 31686 ] {} <Trace> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:33.420050 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.processors_profile_log' in ForegroundLoad
2026.05.06 08:48:33.420170 [ 31696 ] {} <Debug> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading 54 outdated data parts asynchronously
2026.05.06 08:48:33.420211 [ 32361 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_131_131_0 from disk default
2026.05.06 08:48:33.420242 [ 31697 ] {} <Debug> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f) (MergerMutator): Selected 5 parts from 202605_1_171_34 to 202605_175_175_0
2026.05.06 08:48:33.420186 [ 31694 ] {} <Debug> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading 48 outdated data parts asynchronously
2026.05.06 08:48:33.420241 [ 32382 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.asynchronous_metric_log' in ForegroundLoad
2026.05.06 08:48:33.420366 [ 32396 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loading Outdated part 202605_20_20_0 from disk default
2026.05.06 08:48:33.420347 [ 31686 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.53 GiB.
2026.05.06 08:48:33.420499 [ 32373 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loading Outdated part 202605_18_18_0 from disk default
2026.05.06 08:48:33.420477 [ 32398 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_129_129_0 from disk default
2026.05.06 08:48:33.420390 [ 32336 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_130_130_0 from disk default
2026.05.06 08:48:33.420362 [ 32368 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loading Outdated part 202605_19_19_0 from disk default
2026.05.06 08:48:33.420691 [ 32397 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loading Outdated part 202605_17_17_0 from disk default
2026.05.06 08:48:33.420714 [ 31697 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:33.420891 [ 32340 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_166_166_0 from disk default
2026.05.06 08:48:33.420714 [ 32403 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.filesystem_cache_log' in ForegroundLoad
2026.05.06 08:48:33.420977 [ 32351 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_171_171_0 from disk default
2026.05.06 08:48:33.420999 [ 32387 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loading Outdated part 202605_16_16_0 from disk default
2026.05.06 08:48:33.420761 [ 31698 ] {} <Debug> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading 48 outdated data parts asynchronously
2026.05.06 08:48:33.420966 [ 31701 ] {} <Debug> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading 61 outdated data parts asynchronously
2026.05.06 08:48:33.421148 [ 31703 ] {} <Debug> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading 50 outdated data parts asynchronously
2026.05.06 08:48:33.421215 [ 32400 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_170_170_0 from disk default
2026.05.06 08:48:33.421343 [ 32370 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_128_128_0 from disk default
2026.05.06 08:48:33.421388 [ 32337 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_169_169_0 from disk default
2026.05.06 08:48:33.421102 [ 32367 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_165_165_0 from disk default
2026.05.06 08:48:33.421164 [ 32390 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.query_thread_log' in ForegroundLoad
2026.05.06 08:48:33.421625 [ 32359 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_164_164_0 from disk default
2026.05.06 08:48:33.421763 [ 32365 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_127_127_0 from disk default
2026.05.06 08:48:33.421801 [ 31706 ] {} <Debug> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading 54 outdated data parts asynchronously
2026.05.06 08:48:33.421982 [ 32349 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loading Outdated part 202605_1_15_3 from disk default
2026.05.06 08:48:33.422617 [ 32347 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.zookeeper_log' in ForegroundLoad
2026.05.06 08:48:33.422707 [ 32388 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.text_log' in ForegroundLoad
2026.05.06 08:48:33.423072 [ 32360 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_163_163_0 from disk default
2026.05.06 08:48:33.423151 [ 32339 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_162_162_0 from disk default
2026.05.06 08:48:33.423227 [ 32395 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_160_160_0 from disk default
2026.05.06 08:48:33.423249 [ 32383 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.backup_log' with status OK
2026.05.06 08:48:33.423166 [ 32374 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_161_161_0 from disk default
2026.05.06 08:48:33.423248 [ 32338 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_159_159_0 from disk default
2026.05.06 08:48:33.423357 [ 32383 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.crash_log' in ForegroundLoad
2026.05.06 08:48:33.423405 [ 32404 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_158_158_0 from disk default
2026.05.06 08:48:33.423531 [ 32380 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_150_150_0 from disk default
2026.05.06 08:48:33.423747 [ 31702 ] {} <Debug> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383) (MergerMutator): Selected 4 parts from 202605_151_151_0 to 202605_154_154_0
2026.05.06 08:48:33.423780 [ 32401 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_149_149_0 from disk default
2026.05.06 08:48:33.423814 [ 32341 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_148_148_0 from disk default
2026.05.06 08:48:33.423866 [ 31702 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:33.423910 [ 31711 ] {} <Debug> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading 57 outdated data parts asynchronously
2026.05.06 08:48:33.424006 [ 31704 ] {} <Debug> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf) (MergerMutator): Selected 5 parts from 202605_1_181_36 to 202605_185_185_0
2026.05.06 08:48:33.424238 [ 31704 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:33.422876 [ 31697 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.53 GiB.
2026.05.06 08:48:33.424988 [ 32402 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_147_147_0 from disk default
2026.05.06 08:48:33.425555 [ 32392 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_168_168_0 from disk default
2026.05.06 08:48:33.424473 [ 31710 ] {} <Debug> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading 91 outdated data parts asynchronously
2026.05.06 08:48:33.426197 [ 31702 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.53 GiB.
2026.05.06 08:48:33.426200 [ 31700 ] {} <Debug> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa) (MergerMutator): Selected 5 parts from 202605_171_171_0 to 202605_175_175_0
2026.05.06 08:48:33.427140 [ 32379 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_181_181_0 from disk default
2026.05.06 08:48:33.428413 [ 31704 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.53 GiB.
2026.05.06 08:48:33.428566 [ 32385 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_180_180_0 from disk default
2026.05.06 08:48:33.428516 [ 31709 ] {} <Debug> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185) (MergerMutator): Selected 4 parts from 202605_167_167_0 to 202605_170_170_0
2026.05.06 08:48:33.429018 [ 31700 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:33.430205 [ 32376 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_179_179_0 from disk default
2026.05.06 08:48:33.430339 [ 32394 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.opentelemetry_span_log' in ForegroundLoad
2026.05.06 08:48:33.430470 [ 32399 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.query_views_log' with status OK
2026.05.06 08:48:33.430599 [ 32399 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.blob_storage_log' in ForegroundLoad
2026.05.06 08:48:33.428611 [ 31695 ] {} <Debug> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d) (MergerMutator): Selected 5 parts from 202605_301_306_1 to 202605_310_310_0
2026.05.06 08:48:33.439868 [ 31709 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:33.439909 [ 32271 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Debug> bool DB::(anonymous namespace)::checkPermissionsImpl(): Code: 412. DB::Exception: Can't receive Netlink response: error -2. (NETLINK_ERROR) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:33.439997 [ 31700 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.53 GiB.
2026.05.06 08:48:33.440075 [ 32353 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.metric_log' with status OK
2026.05.06 08:48:33.440080 [ 31718 ] {} <Debug> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading 57 outdated data parts asynchronously
2026.05.06 08:48:33.440209 [ 32353 ] {} <Debug> AsyncLoader: Execute load job 'startup table system.query_log' in ForegroundLoad
2026.05.06 08:48:33.441301 [ 32353 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.query_log' with status OK
2026.05.06 08:48:33.443736 [ 32381 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_170_170_0 from disk default
2026.05.06 08:48:33.447899 [ 31695 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:33.448728 [ 32353 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.455825 [ 32342 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.455972 [ 32393 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.transactions_info_log' with status OK
2026.05.06 08:48:33.456026 [ 32393 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.456101 [ 32369 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.error_log' with status OK
2026.05.06 08:48:33.456174 [ 32369 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.456285 [ 32391 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.part_log' with status OK
2026.05.06 08:48:33.456383 [ 32391 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.448278 [ 31709 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.53 GiB.
2026.05.06 08:48:33.448496 [ 31721 ] {} <Debug> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading 11 outdated data parts asynchronously
2026.05.06 08:48:33.463950 [ 32384 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.trace_log' with status OK
2026.05.06 08:48:33.464060 [ 32384 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.464176 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.processors_profile_log' with status OK
2026.05.06 08:48:33.464277 [ 32371 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.464388 [ 32382 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.asynchronous_metric_log' with status OK
2026.05.06 08:48:33.464449 [ 32382 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.455756 [ 31720 ] {} <Debug> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading 48 outdated data parts asynchronously
2026.05.06 08:48:33.464555 [ 32403 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.filesystem_cache_log' with status OK
2026.05.06 08:48:33.472299 [ 31695 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.53 GiB.
2026.05.06 08:48:33.472442 [ 32403 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.473508 [ 32390 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.query_thread_log' with status OK
2026.05.06 08:48:33.473558 [ 32390 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.473630 [ 32347 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.zookeeper_log' with status OK
2026.05.06 08:48:33.473665 [ 32347 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.473722 [ 32388 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.text_log' with status OK
2026.05.06 08:48:33.473754 [ 32388 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.473832 [ 32383 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.crash_log' with status OK
2026.05.06 08:48:33.473922 [ 32383 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.482298 [ 32394 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.opentelemetry_span_log' with status OK
2026.05.06 08:48:33.482353 [ 32394 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.482417 [ 32399 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.blob_storage_log' with status OK
2026.05.06 08:48:33.482447 [ 32399 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.482509 [ 32378 ] {} <Debug> AsyncLoader: Finish load job 'startup table system.session_log' with status OK
2026.05.06 08:48:33.482568 [ 32378 ] {} <Debug> AsyncLoader: Spawn loader worker #2 in ForegroundLoad
2026.05.06 08:48:33.482697 [ 32378 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database system' in ForegroundLoad
2026.05.06 08:48:33.482739 [ 32378 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database system' with status OK
2026.05.06 08:48:33.482766 [ 32378 ] {} <Debug> AsyncLoader: Spawn loader worker #3 in ForegroundLoad
2026.05.06 08:48:33.482821 [ 32378 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database system' in ForegroundLoad
2026.05.06 08:48:33.493335 [ 32344 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.493453 [ 32377 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.493542 [ 31633 ] {} <Debug> AsyncLoader: Wait load job 'startup Atomic database system' in ForegroundLoad
2026.05.06 08:48:33.497301 [ 32378 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database system' with status OK
2026.05.06 08:48:33.497340 [ 32378 ] {} <Debug> AsyncLoader: Stop worker in ForegroundLoad
2026.05.06 08:48:33.497375 [ 32378 ] {} <Debug> AsyncLoader: Change current priority: 0 -> none
2026.05.06 08:48:33.498790 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Debug> MergeTask::PrepareStage: Merging 5 parts: from 202605_1_171_34 to 202605_175_175_0 into Compact with storage Full
2026.05.06 08:48:33.498790 [ 32280 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Debug> MergeTask::PrepareStage: Merging 4 parts: from 202605_167_167_0 to 202605_170_170_0 into Compact with storage Full
2026.05.06 08:48:33.501168 [ 32271 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Debug> MergeTask::PrepareStage: Merging 3 parts: from 202605_1_1_0 to 202605_3_3_0 into Compact with storage Full
2026.05.06 08:48:33.502159 [ 31633 ] {} <Debug> SystemLog: Creating system.query_log from query_log
2026.05.06 08:48:33.505106 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Debug> MergeTask::PrepareStage: Merging 5 parts: from 202605_301_306_1 to 202605_310_310_0 into Compact with storage Full
2026.05.06 08:48:33.505227 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Debug> MergeTask::PrepareStage: Merging 4 parts: from 202605_151_151_0 to 202605_154_154_0 into Compact with storage Full
2026.05.06 08:48:33.500140 [ 32275 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Debug> MergeTask::PrepareStage: Merging 5 parts: from 202605_1_181_36 to 202605_185_185_0 into Wide with storage Full
2026.05.06 08:48:33.500699 [ 32278 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Debug> MergeTask::PrepareStage: Merging 5 parts: from 202605_171_171_0 to 202605_175_175_0 into Compact with storage Full
2026.05.06 08:48:33.530214 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:33.531961 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:33.532116 [ 32275 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:33.532399 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_1_171_34, total 6961 rows starting from the beginning of the part
2026.05.06 08:48:33.532424 [ 32271 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:33.533212 [ 32275 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Debug> MergeTreeSequentialSource: Reading 64 marks from part 202605_1_181_36, total 512740 rows starting from the beginning of the part
2026.05.06 08:48:33.534002 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_301_306_1, total 1372 rows starting from the beginning of the part
2026.05.06 08:48:33.534931 [ 32340 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_166_166_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.534959 [ 32278 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:33.534985 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:33.535028 [ 32380 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_150_150_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535068 [ 32374 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_161_161_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.534936 [ 32392 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_168_168_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535143 [ 32381 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_170_170_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535152 [ 32404 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_158_158_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535211 [ 32339 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_162_162_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535300 [ 32278 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_171_171_0, total 42 rows starting from the beginning of the part
2026.05.06 08:48:33.535338 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_151_151_0, total 433 rows starting from the beginning of the part
2026.05.06 08:48:33.535150 [ 32385 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_180_180_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.534931 [ 32396 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): loadDataPart: inserting 202605_20_20_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535559 [ 32367 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_165_165_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535593 [ 32376 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_179_179_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535631 [ 32373 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): loadDataPart: inserting 202605_18_18_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535678 [ 32395 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_160_160_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535683 [ 32349 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): loadDataPart: inserting 202605_1_15_3 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535754 [ 32397 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): loadDataPart: inserting 202605_17_17_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535780 [ 32280 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:33.535796 [ 32368 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): loadDataPart: inserting 202605_19_19_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535835 [ 32387 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): loadDataPart: inserting 202605_16_16_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.536311 [ 32401 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_149_149_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.536803 [ 32400 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_170_170_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.536895 [ 32402 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_147_147_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.535997 [ 32271 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_1_1_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:33.535714 [ 32367 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_165_165_0 on disk default
2026.05.06 08:48:33.535811 [ 32339 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_162_162_0 on disk default
2026.05.06 08:48:33.535823 [ 32376 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_179_179_0 on disk default
2026.05.06 08:48:33.538013 [ 32379 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_181_181_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.538052 [ 32379 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_181_181_0 on disk default
2026.05.06 08:48:33.535714 [ 32404 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_158_158_0 on disk default
2026.05.06 08:48:33.535872 [ 32387 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Finished loading Outdated part 202605_16_16_0 on disk default
2026.05.06 08:48:33.535715 [ 32381 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_170_170_0 on disk default
2026.05.06 08:48:33.535730 [ 32374 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_161_161_0 on disk default
2026.05.06 08:48:33.535742 [ 32396 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Finished loading Outdated part 202605_20_20_0 on disk default
2026.05.06 08:48:33.535714 [ 32373 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Finished loading Outdated part 202605_18_18_0 on disk default
2026.05.06 08:48:33.538270 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_165_165_0
2026.05.06 08:48:33.538272 [ 32362 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.535766 [ 32380 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_150_150_0 on disk default
2026.05.06 08:48:33.535752 [ 32349 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Finished loading Outdated part 202605_1_15_3 on disk default
2026.05.06 08:48:33.538613 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_162_162_0
2026.05.06 08:48:33.538632 [ 32404 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_158_158_0
2026.05.06 08:48:33.535794 [ 32397 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Finished loading Outdated part 202605_17_17_0 on disk default
2026.05.06 08:48:33.535758 [ 32340 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_166_166_0 on disk default
2026.05.06 08:48:33.536842 [ 32400 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_170_170_0 on disk default
2026.05.06 08:48:33.538693 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04), part: 202605_1_15_3
2026.05.06 08:48:33.536926 [ 32402 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_147_147_0 on disk default
2026.05.06 08:48:33.538737 [ 32397 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04), part: 202605_17_17_0
2026.05.06 08:48:33.536053 [ 32395 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_160_160_0 on disk default
2026.05.06 08:48:33.536478 [ 32392 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_168_168_0 on disk default
2026.05.06 08:48:33.535833 [ 32368 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Finished loading Outdated part 202605_19_19_0 on disk default
2026.05.06 08:48:33.538754 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_166_166_0
2026.05.06 08:48:33.538923 [ 32353 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_178_178_0 from disk default
2026.05.06 08:48:33.539063 [ 32337 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_169_169_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.539106 [ 32337 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_169_169_0 on disk default
2026.05.06 08:48:33.535751 [ 32385 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_180_180_0 on disk default
2026.05.06 08:48:33.539124 [ 32341 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_148_148_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.539168 [ 32341 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_148_148_0 on disk default
2026.05.06 08:48:33.539189 [ 32359 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_164_164_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.539198 [ 32404 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_125_125_0 from disk default
2026.05.06 08:48:33.539186 [ 32343 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_126_126_0 from disk default
2026.05.06 08:48:33.539237 [ 32359 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_164_164_0 on disk default
2026.05.06 08:48:33.539283 [ 32402 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_147_147_0
2026.05.06 08:48:33.539290 [ 32376 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_179_179_0
2026.05.06 08:48:33.539509 [ 32275 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_182_182_0, total 2863 rows starting from the beginning of the part
2026.05.06 08:48:33.536352 [ 32401 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_149_149_0 on disk default
2026.05.06 08:48:33.539551 [ 32379 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_181_181_0
2026.05.06 08:48:33.539551 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_150_150_0
2026.05.06 08:48:33.539584 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_172_172_0, total 36 rows starting from the beginning of the part
2026.05.06 08:48:33.539637 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_170_170_0
2026.05.06 08:48:33.539683 [ 32391 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.539694 [ 32393 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.539706 [ 32384 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.539846 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_160_160_0
2026.05.06 08:48:33.539949 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_170_170_0
2026.05.06 08:48:33.540154 [ 32402 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_153_153_0 from disk default
2026.05.06 08:48:33.540199 [ 32338 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_159_159_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.540240 [ 32338 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_159_159_0 on disk default
2026.05.06 08:48:33.540273 [ 32374 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_161_161_0
2026.05.06 08:48:33.540354 [ 32351 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_171_171_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.540368 [ 32382 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_306_306_0 from disk default
2026.05.06 08:48:33.540398 [ 32396 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04), part: 202605_20_20_0
2026.05.06 08:48:33.540409 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04), part: 202605_18_18_0
2026.05.06 08:48:33.540405 [ 32351 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_171_171_0 on disk default
2026.05.06 08:48:33.540401 [ 32403 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_154_154_0 from disk default
2026.05.06 08:48:33.540465 [ 32384 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.540515 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_168_168_0
2026.05.06 08:48:33.540575 [ 32280 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_167_167_0, total 103 rows starting from the beginning of the part
2026.05.06 08:48:33.540794 [ 32278 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_172_172_0, total 326 rows starting from the beginning of the part
2026.05.06 08:48:33.540807 [ 32271 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_2_2_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:33.540936 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_152_152_0, total 525 rows starting from the beginning of the part
2026.05.06 08:48:33.541402 [ 32362 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.541534 [ 32391 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.541556 [ 32393 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.541838 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_153_153_0, total 785 rows starting from the beginning of the part
2026.05.06 08:48:33.542103 [ 32278 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_173_173_0, total 92 rows starting from the beginning of the part
2026.05.06 08:48:33.542134 [ 32280 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_168_168_0, total 133 rows starting from the beginning of the part
2026.05.06 08:48:33.542357 [ 32384 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.542599 [ 32362 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.542663 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_154_154_0, total 342 rows starting from the beginning of the part
2026.05.06 08:48:33.542930 [ 32391 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.543155 [ 32393 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.543172 [ 32278 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_174_174_0, total 164 rows starting from the beginning of the part
2026.05.06 08:48:33.543303 [ 32390 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_153_153_0 from disk default
2026.05.06 08:48:33.543378 [ 32372 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_152_152_0 from disk default
2026.05.06 08:48:33.543415 [ 32368 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04), part: 202605_19_19_0
2026.05.06 08:48:33.543423 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_159_159_0
2026.05.06 08:48:33.543432 [ 32337 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_169_169_0
2026.05.06 08:48:33.543454 [ 32388 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_169_169_0 from disk default
2026.05.06 08:48:33.543529 [ 32400 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_168_168_0 from disk default
2026.05.06 08:48:33.543655 [ 32394 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_167_167_0 from disk default
2026.05.06 08:48:33.543735 [ 32347 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_124_124_0 from disk default
2026.05.06 08:48:33.543826 [ 32280 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_169_169_0, total 90 rows starting from the beginning of the part
2026.05.06 08:48:33.543931 [ 32384 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.543967 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_307_307_0, total 230 rows starting from the beginning of the part
2026.05.06 08:48:33.544240 [ 32278 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_175_175_0, total 134 rows starting from the beginning of the part
2026.05.06 08:48:33.544566 [ 32360 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_163_163_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.544610 [ 32360 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_163_163_0 on disk default
2026.05.06 08:48:33.545049 [ 32382 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_306_306_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.545090 [ 32382 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_306_306_0 on disk default
2026.05.06 08:48:33.545210 [ 32385 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_180_180_0
2026.05.06 08:48:33.545350 [ 32385 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_123_123_0 from disk default
2026.05.06 08:48:33.545384 [ 32369 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_167_167_0 from disk default
2026.05.06 08:48:33.545431 [ 32341 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_148_148_0
2026.05.06 08:48:33.545621 [ 32395 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_146_146_0 from disk default
2026.05.06 08:48:33.545707 [ 32344 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_305_305_0 from disk default
2026.05.06 08:48:33.547218 [ 32353 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_178_178_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.547259 [ 32353 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_178_178_0 on disk default
2026.05.06 08:48:33.547320 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_178_178_0
2026.05.06 08:48:33.547393 [ 32402 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_153_153_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.547400 [ 32353 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_304_304_0 from disk default
2026.05.06 08:48:33.547501 [ 32367 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_303_303_0 from disk default
2026.05.06 08:48:33.548056 [ 32275 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_183_183_0, total 2863 rows starting from the beginning of the part
2026.05.06 08:48:33.548321 [ 32391 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.548432 [ 32275 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_184_184_0, total 3272 rows starting from the beginning of the part
2026.05.06 08:48:33.548589 [ 32391 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.548706 [ 32362 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.548981 [ 32369 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_167_167_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.549024 [ 32369 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_167_167_0 on disk default
2026.05.06 08:48:33.549077 [ 32275 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_185_185_0, total 2863 rows starting from the beginning of the part
2026.05.06 08:48:33.549091 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_167_167_0
2026.05.06 08:48:33.549337 [ 32393 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.549881 [ 32344 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_305_305_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.549895 [ 32378 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Outdated part 202605_10_10_0 from disk default
2026.05.06 08:48:33.549928 [ 32344 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_305_305_0 on disk default
2026.05.06 08:48:33.549987 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_305_305_0
2026.05.06 08:48:33.549999 [ 32342 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_146_146_0 from disk default
2026.05.06 08:48:33.550114 [ 32337 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_166_166_0 from disk default
2026.05.06 08:48:33.550207 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_145_145_0 from disk default
2026.05.06 08:48:33.550559 [ 32359 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_164_164_0
2026.05.06 08:48:33.550765 [ 32374 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Outdated part 202605_8_8_0 from disk default
2026.05.06 08:48:33.550788 [ 32349 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Outdated part 202605_9_9_0 from disk default
2026.05.06 08:48:33.551556 [ 32395 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_146_146_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.551620 [ 32395 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_146_146_0 on disk default
2026.05.06 08:48:33.551693 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_146_146_0
2026.05.06 08:48:33.551710 [ 32353 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_304_304_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.551785 [ 32353 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_304_304_0 on disk default
2026.05.06 08:48:33.551808 [ 32367 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_303_303_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.551848 [ 32367 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_303_303_0 on disk default
2026.05.06 08:48:33.551872 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_304_304_0
2026.05.06 08:48:33.551867 [ 32395 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Outdated part 202605_7_7_0 from disk default
2026.05.06 08:48:33.551909 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_303_303_0
2026.05.06 08:48:33.552049 [ 32353 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Outdated part 202605_6_6_0 from disk default
2026.05.06 08:48:33.553202 [ 32378 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_10_10_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.553241 [ 32378 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Outdated part 202605_10_10_0 on disk default
2026.05.06 08:48:33.553326 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e), part: 202605_10_10_0
2026.05.06 08:48:33.553328 [ 32342 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_146_146_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.553359 [ 32402 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_153_153_0 on disk default
2026.05.06 08:48:33.553372 [ 32342 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_146_146_0 on disk default
2026.05.06 08:48:33.553437 [ 32337 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_166_166_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.553472 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_146_146_0
2026.05.06 08:48:33.553477 [ 32337 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_166_166_0 on disk default
2026.05.06 08:48:33.553558 [ 32337 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_166_166_0
2026.05.06 08:48:33.553543 [ 32378 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_150_150_0 from disk default
2026.05.06 08:48:33.553727 [ 32383 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_157_157_0 from disk default
2026.05.06 08:48:33.553742 [ 32375 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_302_302_0 from disk default
2026.05.06 08:48:33.553751 [ 32342 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_152_152_0 from disk default
2026.05.06 08:48:33.553960 [ 32374 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_8_8_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.554018 [ 32374 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Outdated part 202605_8_8_0 on disk default
2026.05.06 08:48:33.554029 [ 32349 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_9_9_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.554068 [ 32349 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Outdated part 202605_9_9_0 on disk default
2026.05.06 08:48:33.554078 [ 32374 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e), part: 202605_8_8_0
2026.05.06 08:48:33.553378 [ 32386 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_151_151_0 from disk default
2026.05.06 08:48:33.554157 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e), part: 202605_9_9_0
2026.05.06 08:48:33.554214 [ 32402 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_153_153_0
2026.05.06 08:48:33.554252 [ 32392 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_165_165_0 from disk default
2026.05.06 08:48:33.554310 [ 32402 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_122_122_0 from disk default
2026.05.06 08:48:33.554377 [ 32368 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_151_151_0 from disk default
2026.05.06 08:48:33.554512 [ 32349 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_150_150_0 from disk default
2026.05.06 08:48:33.554895 [ 32351 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_171_171_0
2026.05.06 08:48:33.555089 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04), part: 202605_16_16_0
2026.05.06 08:48:33.555882 [ 32380 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_166_166_0 from disk default
2026.05.06 08:48:33.556024 [ 32376 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_156_156_0 from disk default
2026.05.06 08:48:33.556263 [ 32391 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.556567 [ 32360 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_163_163_0
2026.05.06 08:48:33.556589 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_306_306_0
2026.05.06 08:48:33.556631 [ 32338 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_145_145_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.556673 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_145_145_0 on disk default
2026.05.06 08:48:33.556707 [ 32360 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_121_121_0 from disk default
2026.05.06 08:48:33.556717 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_149_149_0
2026.05.06 08:48:33.556800 [ 32382 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_165_165_0 from disk default
2026.05.06 08:48:33.556990 [ 32401 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_149_149_0 from disk default
2026.05.06 08:48:33.557223 [ 31690 ] {} <Debug> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Loaded 6 outdated data parts asynchronously
2026.05.06 08:48:33.557293 [ 32342 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_152_152_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.557334 [ 32342 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_152_152_0 on disk default
2026.05.06 08:48:33.558109 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_173_173_0, total 36 rows starting from the beginning of the part
2026.05.06 08:48:33.558458 [ 32383 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_157_157_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.558504 [ 32383 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_157_157_0 on disk default
2026.05.06 08:48:33.559021 [ 32395 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_7_7_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.559061 [ 32395 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Outdated part 202605_7_7_0 on disk default
2026.05.06 08:48:33.559453 [ 32362 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.559596 [ 32384 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.560318 [ 32376 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_156_156_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.560369 [ 32376 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_156_156_0 on disk default
2026.05.06 08:48:33.560372 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_157_157_0
2026.05.06 08:48:33.560431 [ 32376 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_156_156_0
2026.05.06 08:48:33.560536 [ 32383 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_145_145_0 from disk default
2026.05.06 08:48:33.560856 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e), part: 202605_7_7_0
2026.05.06 08:48:33.560913 [ 32393 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.560919 [ 32280 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_170_170_0, total 132 rows starting from the beginning of the part
2026.05.06 08:48:33.561221 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_152_152_0
2026.05.06 08:48:33.561238 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_145_145_0
2026.05.06 08:48:33.561307 [ 32388 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_169_169_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.561350 [ 32388 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_169_169_0 on disk default
2026.05.06 08:48:33.561411 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_169_169_0
2026.05.06 08:48:33.561848 [ 32375 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_302_302_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.561891 [ 32375 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_302_302_0 on disk default
2026.05.06 08:48:33.561954 [ 32375 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_302_302_0
2026.05.06 08:48:33.562245 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_174_174_0, total 12 rows starting from the beginning of the part
2026.05.06 08:48:33.562327 [ 32391 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.562669 [ 32362 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.562699 [ 32271 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_3_3_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:33.562956 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_175_175_0, total 13 rows starting from the beginning of the part
2026.05.06 08:48:33.563094 [ 32353 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_6_6_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.563134 [ 32353 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Outdated part 202605_6_6_0 on disk default
2026.05.06 08:48:33.563193 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e), part: 202605_6_6_0
2026.05.06 08:48:33.563235 [ 32384 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.563667 [ 32392 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_165_165_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.563719 [ 32392 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_165_165_0 on disk default
2026.05.06 08:48:33.563786 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_165_165_0
2026.05.06 08:48:33.563791 [ 32393 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.564007 [ 32368 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_151_151_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.564056 [ 32368 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_151_151_0 on disk default
2026.05.06 08:48:33.564117 [ 32368 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_151_151_0
2026.05.06 08:48:33.564216 [ 32396 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_166_166_0 from disk default
2026.05.06 08:48:33.564301 [ 32395 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_149_149_0 from disk default
2026.05.06 08:48:33.564500 [ 32342 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_148_148_0 from disk default
2026.05.06 08:48:33.564597 [ 32392 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_155_155_0 from disk default
2026.05.06 08:48:33.564644 [ 32340 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_154_154_0 from disk default
2026.05.06 08:48:33.564685 [ 32337 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_164_164_0 from disk default
2026.05.06 08:48:33.564775 [ 32368 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_120_120_0 from disk default
2026.05.06 08:48:33.564873 [ 32399 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_165_165_0 from disk default
2026.05.06 08:48:33.564995 [ 32353 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_301_301_0 from disk default
2026.05.06 08:48:33.565366 [ 32394 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_167_167_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.565414 [ 32394 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_167_167_0 on disk default
2026.05.06 08:48:33.565453 [ 32391 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.565861 [ 32401 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_149_149_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.565902 [ 32401 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_149_149_0 on disk default
2026.05.06 08:48:33.566064 [ 32383 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_145_145_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.566105 [ 32383 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_145_145_0 on disk default
2026.05.06 08:48:33.566308 [ 32349 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_150_150_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.566357 [ 32349 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_150_150_0 on disk default
2026.05.06 08:48:33.566367 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_167_167_0
2026.05.06 08:48:33.566515 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_150_150_0
2026.05.06 08:48:33.566524 [ 32394 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_148_148_0 from disk default
2026.05.06 08:48:33.566701 [ 32349 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_144_144_0 from disk default
2026.05.06 08:48:33.566809 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_308_308_0, total 111 rows starting from the beginning of the part
2026.05.06 08:48:33.567322 [ 32340 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_154_154_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.567362 [ 32340 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_154_154_0 on disk default
2026.05.06 08:48:33.567422 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_154_154_0
2026.05.06 08:48:33.567559 [ 32340 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Outdated part 202605_5_5_0 from disk default
2026.05.06 08:48:33.567641 [ 32392 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_155_155_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.567682 [ 32392 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_155_155_0 on disk default
2026.05.06 08:48:33.567925 [ 32362 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.568066 [ 32384 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.568648 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_149_149_0
2026.05.06 08:48:33.568675 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_145_145_0
2026.05.06 08:48:33.569295 [ 32401 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_143_143_0 from disk default
2026.05.06 08:48:33.569408 [ 32383 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_164_164_0 from disk default
2026.05.06 08:48:33.569454 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_155_155_0
2026.05.06 08:48:33.569643 [ 32344 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_300_300_0 from disk default
2026.05.06 08:48:33.570387 [ 32400 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_168_168_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.570429 [ 32400 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_168_168_0 on disk default
2026.05.06 08:48:33.570498 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_168_168_0
2026.05.06 08:48:33.570656 [ 32400 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_163_163_0 from disk default
2026.05.06 08:48:33.571977 [ 32399 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_165_165_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.572031 [ 32399 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_165_165_0 on disk default
2026.05.06 08:48:33.572096 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_165_165_0
2026.05.06 08:48:33.572158 [ 32396 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_166_166_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.572203 [ 32396 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_166_166_0 on disk default
2026.05.06 08:48:33.572237 [ 32399 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_144_144_0 from disk default
2026.05.06 08:48:33.572458 [ 32396 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_166_166_0
2026.05.06 08:48:33.572621 [ 32396 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_162_162_0 from disk default
2026.05.06 08:48:33.572991 [ 32394 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_148_148_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.573036 [ 32394 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_148_148_0 on disk default
2026.05.06 08:48:33.573096 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_148_148_0
2026.05.06 08:48:33.573224 [ 32394 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_143_143_0 from disk default
2026.05.06 08:48:33.573700 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_309_309_0, total 165 rows starting from the beginning of the part
2026.05.06 08:48:33.574399 [ 32383 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_164_164_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.574444 [ 32383 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_164_164_0 on disk default
2026.05.06 08:48:33.574513 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_164_164_0
2026.05.06 08:48:33.574656 [ 32383 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_299_299_0 from disk default
2026.05.06 08:48:33.575421 [ 32393 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.576745 [ 32353 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_301_301_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.576790 [ 32353 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_301_301_0 on disk default
2026.05.06 08:48:33.576856 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_301_301_0
2026.05.06 08:48:33.576977 [ 32369 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_153_153_0 from disk default
2026.05.06 08:48:33.578558 [ 32344 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_300_300_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.578605 [ 32344 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_300_300_0 on disk default
2026.05.06 08:48:33.578833 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_300_300_0
2026.05.06 08:48:33.579022 [ 32344 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_164_164_0 from disk default
2026.05.06 08:48:33.580369 [ 32340 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_5_5_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.580419 [ 32340 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Outdated part 202605_5_5_0 on disk default
2026.05.06 08:48:33.580503 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e), part: 202605_5_5_0
2026.05.06 08:48:33.580642 [ 32340 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_142_142_0 from disk default
2026.05.06 08:48:33.583040 [ 32369 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_153_153_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.583085 [ 32369 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_153_153_0 on disk default
2026.05.06 08:48:33.583144 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_153_153_0
2026.05.06 08:48:33.583317 [ 32375 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_119_119_0 from disk default
2026.05.06 08:48:33.585167 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_310_310_0, total 142 rows starting from the beginning of the part
2026.05.06 08:48:33.585747 [ 32349 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_144_144_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.585798 [ 32349 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_144_144_0 on disk default
2026.05.06 08:48:33.585849 [ 32391 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:33.585868 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_144_144_0
2026.05.06 08:48:33.586010 [ 32349 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_298_298_0 from disk default
2026.05.06 08:48:33.586009 [ 32344 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_164_164_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.586093 [ 32344 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_164_164_0 on disk default
2026.05.06 08:48:33.586163 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_164_164_0
2026.05.06 08:48:33.586281 [ 32344 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_163_163_0 from disk default
2026.05.06 08:48:33.586317 [ 32382 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_165_165_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.586361 [ 32382 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_165_165_0 on disk default
2026.05.06 08:48:33.588619 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_165_165_0
2026.05.06 08:48:33.588758 [ 32382 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_297_297_0 from disk default
2026.05.06 08:48:33.590024 [ 32399 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_144_144_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.590068 [ 32399 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_144_144_0 on disk default
2026.05.06 08:48:33.590093 [ 32390 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_153_153_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.590133 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_144_144_0
2026.05.06 08:48:33.590135 [ 32390 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_153_153_0 on disk default
2026.05.06 08:48:33.590235 [ 32390 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_153_153_0
2026.05.06 08:48:33.590349 [ 32374 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_161_161_0 from disk default
2026.05.06 08:48:33.590547 [ 32390 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Outdated part 202605_4_4_0 from disk default
2026.05.06 08:48:33.592986 [ 32372 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_152_152_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.593028 [ 32372 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_152_152_0 on disk default
2026.05.06 08:48:33.593089 [ 32372 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_152_152_0
2026.05.06 08:48:33.593249 [ 32372 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_162_162_0 from disk default
2026.05.06 08:48:33.593648 [ 32340 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_142_142_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.593696 [ 32340 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_142_142_0 on disk default
2026.05.06 08:48:33.593762 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_142_142_0
2026.05.06 08:48:33.593927 [ 32340 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_160_160_0 from disk default
2026.05.06 08:48:33.594558 [ 32394 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_143_143_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.594603 [ 32394 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_143_143_0 on disk default
2026.05.06 08:48:33.594675 [ 32401 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_143_143_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.594716 [ 32401 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_143_143_0 on disk default
2026.05.06 08:48:33.595811 [ 32380 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_166_166_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.595856 [ 32380 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_166_166_0 on disk default
2026.05.06 08:48:33.597203 [ 32344 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_163_163_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.597248 [ 32344 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_163_163_0 on disk default
2026.05.06 08:48:33.597617 [ 32337 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_164_164_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.597659 [ 32337 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_164_164_0 on disk default
2026.05.06 08:48:33.597668 [ 32390 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_4_4_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.597712 [ 32390 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Outdated part 202605_4_4_0 on disk default
2026.05.06 08:48:33.598051 [ 32396 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_162_162_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.598091 [ 32396 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_162_162_0 on disk default
2026.05.06 08:48:33.599317 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_143_143_0
2026.05.06 08:48:33.599326 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_143_143_0
2026.05.06 08:48:33.599356 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_166_166_0
2026.05.06 08:48:33.599374 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_163_163_0
2026.05.06 08:48:33.599559 [ 32337 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_164_164_0
2026.05.06 08:48:33.600032 [ 32390 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e), part: 202605_4_4_0
2026.05.06 08:48:33.600043 [ 32396 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_162_162_0
2026.05.06 08:48:33.600173 [ 32396 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_142_142_0 from disk default
2026.05.06 08:48:33.600347 [ 32380 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_177_177_0 from disk default
2026.05.06 08:48:33.600414 [ 32388 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_296_296_0 from disk default
2026.05.06 08:48:33.600455 [ 32351 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_141_141_0 from disk default
2026.05.06 08:48:33.600541 [ 32392 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Outdated part 202605_3_3_0 from disk default
2026.05.06 08:48:33.600616 [ 32339 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_152_152_0 from disk default
2026.05.06 08:48:33.600624 [ 32376 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_163_163_0 from disk default
2026.05.06 08:48:33.602279 [ 32380 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_177_177_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.602325 [ 32380 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_177_177_0 on disk default
2026.05.06 08:48:33.602385 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_177_177_0
2026.05.06 08:48:33.602516 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_161_161_0 from disk default
2026.05.06 08:48:33.605031 [ 32339 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_152_152_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.605085 [ 32339 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_152_152_0 on disk default
2026.05.06 08:48:33.605158 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_152_152_0
2026.05.06 08:48:33.605294 [ 32387 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_141_141_0 from disk default
2026.05.06 08:48:33.608377 [ 32380 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_161_161_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.608423 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_161_161_0 on disk default
2026.05.06 08:48:33.608541 [ 32351 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_141_141_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.608586 [ 32351 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_141_141_0 on disk default
2026.05.06 08:48:33.609100 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_161_161_0
2026.05.06 08:48:33.609248 [ 32380 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Outdated part 202605_2_2_0 from disk default
2026.05.06 08:48:33.610230 [ 32351 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_141_141_0
2026.05.06 08:48:33.610385 [ 32351 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_162_162_0 from disk default
2026.05.06 08:48:33.611430 [ 32349 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_298_298_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.611498 [ 32349 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_298_298_0 on disk default
2026.05.06 08:48:33.611510 [ 32383 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_299_299_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.611555 [ 32383 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_299_299_0 on disk default
2026.05.06 08:48:33.611564 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_298_298_0
2026.05.06 08:48:33.611631 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_299_299_0
2026.05.06 08:48:33.611773 [ 32381 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_140_140_0 from disk default
2026.05.06 08:48:33.612558 [ 32383 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_139_139_0 from disk default
2026.05.06 08:48:33.615860 [ 32403 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_154_154_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.615915 [ 32403 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_154_154_0 on disk default
2026.05.06 08:48:33.616002 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_154_154_0
2026.05.06 08:48:33.616155 [ 32403 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_140_140_0 from disk default
2026.05.06 08:48:33.617610 [ 32378 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_150_150_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.617653 [ 32378 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_150_150_0 on disk default
2026.05.06 08:48:33.617740 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_150_150_0
2026.05.06 08:48:33.617953 [ 32392 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_3_3_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.617998 [ 32392 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Outdated part 202605_3_3_0 on disk default
2026.05.06 08:48:33.618098 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e), part: 202605_3_3_0
2026.05.06 08:48:33.618651 [ 32399 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Outdated part 202605_1_1_0 from disk default
2026.05.06 08:48:33.618786 [ 32392 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loading Outdated part 202605_1_6_1 from disk default
2026.05.06 08:48:33.620367 [ 32380 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_2_2_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.620419 [ 32380 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Outdated part 202605_2_2_0 on disk default
2026.05.06 08:48:33.620508 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e), part: 202605_2_2_0
2026.05.06 08:48:33.620604 [ 32395 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_149_149_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.620646 [ 32395 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_149_149_0 on disk default
2026.05.06 08:48:33.620707 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_149_149_0
2026.05.06 08:48:33.620723 [ 32380 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_138_138_0 from disk default
2026.05.06 08:48:33.621667 [ 32379 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_295_295_0 from disk default
2026.05.06 08:48:33.624594 [ 32340 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_160_160_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.624638 [ 32340 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_160_160_0 on disk default
2026.05.06 08:48:33.624699 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_160_160_0
2026.05.06 08:48:33.624829 [ 32340 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_159_159_0 from disk default
2026.05.06 08:48:33.627907 [ 31633 ] {} <Debug> SystemLog: Creating system.query_thread_log from query_thread_log
2026.05.06 08:48:33.630260 [ 32342 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_148_148_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.630305 [ 32342 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_148_148_0 on disk default
2026.05.06 08:48:33.630338 [ 32386 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_151_151_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.630371 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_148_148_0
2026.05.06 08:48:33.630379 [ 32386 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_151_151_0 on disk default
2026.05.06 08:48:33.630447 [ 32386 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_151_151_0
2026.05.06 08:48:33.630517 [ 32342 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_294_294_0 from disk default
2026.05.06 08:48:33.630584 [ 32386 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_139_139_0 from disk default
2026.05.06 08:48:33.630685 [ 32379 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_295_295_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.630730 [ 32379 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_295_295_0 on disk default
2026.05.06 08:48:33.630793 [ 32379 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_295_295_0
2026.05.06 08:48:33.630939 [ 32379 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_147_147_0 from disk default
2026.05.06 08:48:33.632924 [ 32380 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_138_138_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.632996 [ 32380 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_138_138_0 on disk default
2026.05.06 08:48:33.633089 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_138_138_0
2026.05.06 08:48:33.633234 [ 32371 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_176_176_0 from disk default
2026.05.06 08:48:33.634133 [ 32379 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_147_147_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.634177 [ 32379 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_147_147_0 on disk default
2026.05.06 08:48:33.634238 [ 32379 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_147_147_0
2026.05.06 08:48:33.634353 [ 32379 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_160_160_0 from disk default
2026.05.06 08:48:33.635169 [ 32371 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_176_176_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.635208 [ 32371 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_176_176_0 on disk default
2026.05.06 08:48:33.635916 [ 32371 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_176_176_0
2026.05.06 08:48:33.636133 [ 32344 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_151_151_0 from disk default
2026.05.06 08:48:33.637213 [ 32340 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_159_159_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.637257 [ 32340 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_159_159_0 on disk default
2026.05.06 08:48:33.637320 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_159_159_0
2026.05.06 08:48:33.637500 [ 32337 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_147_147_0 from disk default
2026.05.06 08:48:33.638232 [ 32386 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_139_139_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.638276 [ 32386 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_139_139_0 on disk default
2026.05.06 08:48:33.638340 [ 32386 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_139_139_0
2026.05.06 08:48:33.638478 [ 32386 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_161_161_0 from disk default
2026.05.06 08:48:33.638720 [ 32344 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_151_151_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.638770 [ 32344 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_151_151_0 on disk default
2026.05.06 08:48:33.638826 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_151_151_0
2026.05.06 08:48:33.638990 [ 32344 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_146_146_0 from disk default
2026.05.06 08:48:33.642527 [ 32342 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_294_294_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.642577 [ 32342 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_294_294_0 on disk default
2026.05.06 08:48:33.642643 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_294_294_0
2026.05.06 08:48:33.642773 [ 32342 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_158_158_0 from disk default
2026.05.06 08:48:33.643334 [ 32379 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_160_160_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.643381 [ 32379 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_160_160_0 on disk default
2026.05.06 08:48:33.643835 [ 32379 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_160_160_0
2026.05.06 08:48:33.643976 [ 32379 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_137_137_0 from disk default
2026.05.06 08:48:33.645219 [ 32386 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_161_161_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.645266 [ 32386 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_161_161_0 on disk default
2026.05.06 08:48:33.645327 [ 32386 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_161_161_0
2026.05.06 08:48:33.645498 [ 32349 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_159_159_0 from disk default
2026.05.06 08:48:33.649351 [ 32349 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_159_159_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.649397 [ 32349 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_159_159_0 on disk default
2026.05.06 08:48:33.649459 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_159_159_0
2026.05.06 08:48:33.649607 [ 32349 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_145_145_0 from disk default
2026.05.06 08:48:33.652313 [ 32372 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_162_162_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.652360 [ 32372 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_162_162_0 on disk default
2026.05.06 08:48:33.652420 [ 32372 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_162_162_0
2026.05.06 08:48:33.652562 [ 32372 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_157_157_0 from disk default
2026.05.06 08:48:33.652596 [ 32400 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_163_163_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.652638 [ 32400 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_163_163_0 on disk default
2026.05.06 08:48:33.652662 [ 32396 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_142_142_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.652698 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_163_163_0
2026.05.06 08:48:33.652714 [ 32396 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_142_142_0 on disk default
2026.05.06 08:48:33.652796 [ 32382 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_297_297_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.652844 [ 32382 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_297_297_0 on disk default
2026.05.06 08:48:33.652893 [ 32400 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_146_146_0 from disk default
2026.05.06 08:48:33.653088 [ 32399 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_1_1_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.653155 [ 32399 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Outdated part 202605_1_1_0 on disk default
2026.05.06 08:48:33.653165 [ 32392 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): loadDataPart: inserting 202605_1_6_1 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.653210 [ 32392 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Finished loading Outdated part 202605_1_6_1 on disk default
2026.05.06 08:48:33.653243 [ 32387 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_141_141_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.653272 [ 32351 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_162_162_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.653293 [ 32387 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_141_141_0 on disk default
2026.05.06 08:48:33.653332 [ 32351 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_162_162_0 on disk default
2026.05.06 08:48:33.654718 [ 32374 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_161_161_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.654771 [ 32374 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_161_161_0 on disk default
2026.05.06 08:48:33.655284 [ 32396 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_142_142_0
2026.05.06 08:48:33.655319 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_297_297_0
2026.05.06 08:48:33.655364 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e), part: 202605_1_1_0
2026.05.06 08:48:33.655381 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e), part: 202605_1_6_1
2026.05.06 08:48:33.655403 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_141_141_0
2026.05.06 08:48:33.655421 [ 32351 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_162_162_0
2026.05.06 08:48:33.655440 [ 32396 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_175_175_0 from disk default
2026.05.06 08:48:33.655428 [ 32374 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_161_161_0
2026.05.06 08:48:33.655782 [ 32399 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_145_145_0 from disk default
2026.05.06 08:48:33.655855 [ 32382 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_144_144_0 from disk default
2026.05.06 08:48:33.655982 [ 32387 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_160_160_0 from disk default
2026.05.06 08:48:33.656065 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_293_293_0 from disk default
2026.05.06 08:48:33.656208 [ 32351 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_118_118_0 from disk default
2026.05.06 08:48:33.657488 [ 32379 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_137_137_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.657531 [ 32379 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_137_137_0 on disk default
2026.05.06 08:48:33.657817 [ 32379 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_137_137_0
2026.05.06 08:48:33.657923 [ 31721 ] {} <Debug> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Loaded 11 outdated data parts asynchronously
2026.05.06 08:48:33.657938 [ 32359 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_159_159_0 from disk default
2026.05.06 08:48:33.660207 [ 32379 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_143_143_0 from disk default
2026.05.06 08:48:33.660771 [ 32396 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_175_175_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.660816 [ 32396 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_175_175_0 on disk default
2026.05.06 08:48:33.660878 [ 32396 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_175_175_0
2026.05.06 08:48:33.660925 [ 32376 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_163_163_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.660981 [ 32376 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_163_163_0 on disk default
2026.05.06 08:48:33.661012 [ 32396 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_117_117_0 from disk default
2026.05.06 08:48:33.662190 [ 32376 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_163_163_0
2026.05.06 08:48:33.662324 [ 32376 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_292_292_0 from disk default
2026.05.06 08:48:33.662040 [ 31721 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 217 ms.
2026.05.06 08:48:33.668170 [ 32400 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_146_146_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.668224 [ 32400 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_146_146_0 on disk default
2026.05.06 08:48:33.668307 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_146_146_0
2026.05.06 08:48:33.668444 [ 32382 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_144_144_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.668471 [ 32400 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_158_158_0 from disk default
2026.05.06 08:48:33.668508 [ 32382 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_144_144_0 on disk default
2026.05.06 08:48:33.668585 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_144_144_0
2026.05.06 08:48:33.668792 [ 32382 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_158_158_0 from disk default
2026.05.06 08:48:33.670868 [ 32381 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_140_140_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.670919 [ 32381 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_140_140_0 on disk default
2026.05.06 08:48:33.670987 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_140_140_0
2026.05.06 08:48:33.671115 [ 32381 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_136_136_0 from disk default
2026.05.06 08:48:33.671789 [ 32359 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_159_159_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.671841 [ 32359 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_159_159_0 on disk default
2026.05.06 08:48:33.672271 [ 32359 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_159_159_0
2026.05.06 08:48:33.672404 [ 32359 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_291_291_0 from disk default
2026.05.06 08:48:33.673208 [ 32388 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_296_296_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.673268 [ 32388 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_296_296_0 on disk default
2026.05.06 08:48:33.673730 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_296_296_0
2026.05.06 08:48:33.673864 [ 32388 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_157_157_0 from disk default
2026.05.06 08:48:33.675781 [ 32403 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_140_140_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.675833 [ 32403 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_140_140_0 on disk default
2026.05.06 08:48:33.675889 [ 32387 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_160_160_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.675901 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_140_140_0
2026.05.06 08:48:33.675931 [ 32379 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_143_143_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.675939 [ 32387 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_160_160_0 on disk default
2026.05.06 08:48:33.675977 [ 32379 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_143_143_0 on disk default
2026.05.06 08:48:33.676009 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_160_160_0
2026.05.06 08:48:33.676038 [ 32403 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_290_290_0 from disk default
2026.05.06 08:48:33.676041 [ 32379 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_143_143_0
2026.05.06 08:48:33.676157 [ 32387 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_138_138_0 from disk default
2026.05.06 08:48:33.676335 [ 32379 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_156_156_0 from disk default
2026.05.06 08:48:33.677812 [ 32399 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_145_145_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.677863 [ 32399 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_145_145_0 on disk default
2026.05.06 08:48:33.679616 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_145_145_0
2026.05.06 08:48:33.679785 [ 32399 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_150_150_0 from disk default
2026.05.06 08:48:33.680562 [ 32401 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_293_293_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.680604 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_293_293_0 on disk default
2026.05.06 08:48:33.680664 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_293_293_0
2026.05.06 08:48:33.680845 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_289_289_0 from disk default
2026.05.06 08:48:33.681098 [ 32400 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_158_158_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.681160 [ 32400 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_158_158_0 on disk default
2026.05.06 08:48:33.681248 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_158_158_0
2026.05.06 08:48:33.681404 [ 32400 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_142_142_0 from disk default
2026.05.06 08:48:33.682950 [ 32383 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_139_139_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.682998 [ 32383 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_139_139_0 on disk default
2026.05.06 08:48:33.683063 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_139_139_0
2026.05.06 08:48:33.683256 [ 32383 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_149_149_0 from disk default
2026.05.06 08:48:33.684748 [ 32376 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_292_292_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.684795 [ 32376 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_292_292_0 on disk default
2026.05.06 08:48:33.684859 [ 32376 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_292_292_0
2026.05.06 08:48:33.685019 [ 32376 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_288_288_0 from disk default
2026.05.06 08:48:33.686531 [ 32359 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_291_291_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.686578 [ 32359 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_291_291_0 on disk default
2026.05.06 08:48:33.686844 [ 32383 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_149_149_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.686899 [ 32383 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_149_149_0 on disk default
2026.05.06 08:48:33.687111 [ 32388 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_157_157_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.687161 [ 32388 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_157_157_0 on disk default
2026.05.06 08:48:33.687168 [ 32381 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_136_136_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.687212 [ 32381 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_136_136_0 on disk default
2026.05.06 08:48:33.688453 [ 32399 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_150_150_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.688523 [ 32399 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_150_150_0 on disk default
2026.05.06 08:48:33.688784 [ 32359 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_291_291_0
2026.05.06 08:48:33.688820 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_149_149_0
2026.05.06 08:48:33.688831 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_157_157_0
2026.05.06 08:48:33.688929 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_136_136_0
2026.05.06 08:48:33.688976 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_150_150_0
2026.05.06 08:48:33.689090 [ 32392 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_135_135_0 from disk default
2026.05.06 08:48:33.689382 [ 32388 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_174_174_0 from disk default
2026.05.06 08:48:33.689456 [ 32359 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_157_157_0 from disk default
2026.05.06 08:48:33.689548 [ 32399 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_137_137_0 from disk default
2026.05.06 08:48:33.689570 [ 32381 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_134_134_0 from disk default
2026.05.06 08:48:33.689737 [ 32342 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_158_158_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.689788 [ 32342 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_158_158_0 on disk default
2026.05.06 08:48:33.689999 [ 32372 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_157_157_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.690002 [ 32382 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_158_158_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.690067 [ 32372 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_157_157_0 on disk default
2026.05.06 08:48:33.690096 [ 32382 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_158_158_0 on disk default
2026.05.06 08:48:33.690588 [ 32403 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_290_290_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.690636 [ 32403 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_290_290_0 on disk default
2026.05.06 08:48:33.690810 [ 32401 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_289_289_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.690863 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_289_289_0 on disk default
2026.05.06 08:48:33.692210 [ 32387 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_138_138_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.692261 [ 32387 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_138_138_0 on disk default
2026.05.06 08:48:33.692422 [ 32400 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_142_142_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.692466 [ 32400 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_142_142_0 on disk default
2026.05.06 08:48:33.692854 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_158_158_0
2026.05.06 08:48:33.692896 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_158_158_0
2026.05.06 08:48:33.692918 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_290_290_0
2026.05.06 08:48:33.692934 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_289_289_0
2026.05.06 08:48:33.692969 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_142_142_0
2026.05.06 08:48:33.692892 [ 32372 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_157_157_0
2026.05.06 08:48:33.692969 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_138_138_0
2026.05.06 08:48:33.692983 [ 32342 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_148_148_0 from disk default
2026.05.06 08:48:33.693058 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_287_287_0 from disk default
2026.05.06 08:48:33.693144 [ 32353 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_141_141_0 from disk default
2026.05.06 08:48:33.693196 [ 32403 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_136_136_0 from disk default
2026.05.06 08:48:33.693312 [ 32400 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_133_133_0 from disk default
2026.05.06 08:48:33.693387 [ 32382 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_156_156_0 from disk default
2026.05.06 08:48:33.693573 [ 32372 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_147_147_0 from disk default
2026.05.06 08:48:33.695190 [ 32342 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_148_148_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.695236 [ 32342 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_148_148_0 on disk default
2026.05.06 08:48:33.695753 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_148_148_0
2026.05.06 08:48:33.695916 [ 32387 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_286_286_0 from disk default
2026.05.06 08:48:33.696640 [ 32353 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_141_141_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.696687 [ 32353 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_141_141_0 on disk default
2026.05.06 08:48:33.696752 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_141_141_0
2026.05.06 08:48:33.696876 [ 32353 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_156_156_0 from disk default
2026.05.06 08:48:33.698691 [ 32388 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_174_174_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.698734 [ 32388 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_174_174_0 on disk default
2026.05.06 08:48:33.698796 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_174_174_0
2026.05.06 08:48:33.698936 [ 32388 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_132_132_0 from disk default
2026.05.06 08:48:33.699386 [ 32337 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_147_147_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.699444 [ 32337 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_147_147_0 on disk default
2026.05.06 08:48:33.700918 [ 32337 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_147_147_0
2026.05.06 08:48:33.701135 [ 32377 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_285_285_0 from disk default
2026.05.06 08:48:33.703505 [ 32372 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_147_147_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.703566 [ 32372 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_147_147_0 on disk default
2026.05.06 08:48:33.703657 [ 32372 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_147_147_0
2026.05.06 08:48:33.703832 [ 32372 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_140_140_0 from disk default
2026.05.06 08:48:33.705773 [ 32376 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_288_288_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.705821 [ 32376 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_288_288_0 on disk default
2026.05.06 08:48:33.705887 [ 32376 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_288_288_0
2026.05.06 08:48:33.706018 [ 32376 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_155_155_0 from disk default
2026.05.06 08:48:33.707580 [ 32403 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_136_136_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.707721 [ 32403 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_136_136_0 on disk default
2026.05.06 08:48:33.707797 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_136_136_0
2026.05.06 08:48:33.707949 [ 32403 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_155_155_0 from disk default
2026.05.06 08:48:33.709718 [ 32344 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_146_146_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.709770 [ 32344 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_146_146_0 on disk default
2026.05.06 08:48:33.709842 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_146_146_0
2026.05.06 08:48:33.709863 [ 32359 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_157_157_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.709927 [ 32359 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_157_157_0 on disk default
2026.05.06 08:48:33.709996 [ 32344 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_155_155_0 from disk default
2026.05.06 08:48:33.710244 [ 32399 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_137_137_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.710292 [ 32399 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_137_137_0 on disk default
2026.05.06 08:48:33.710384 [ 32392 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_135_135_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.710431 [ 32392 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_135_135_0 on disk default
2026.05.06 08:48:33.710887 [ 32379 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_156_156_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.710941 [ 32379 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_156_156_0 on disk default
2026.05.06 08:48:33.711927 [ 32353 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_156_156_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.711926 [ 32377 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_285_285_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.711985 [ 32353 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_156_156_0 on disk default
2026.05.06 08:48:33.712021 [ 32377 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_285_285_0 on disk default
2026.05.06 08:48:33.712127 [ 32359 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_157_157_0
2026.05.06 08:48:33.712264 [ 32359 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_154_154_0 from disk default
2026.05.06 08:48:33.712299 [ 32401 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_287_287_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.712367 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_287_287_0 on disk default
2026.05.06 08:48:33.712381 [ 32387 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_286_286_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.712433 [ 32387 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_286_286_0 on disk default
2026.05.06 08:48:33.712508 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_286_286_0
2026.05.06 08:48:33.712538 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_137_137_0
2026.05.06 08:48:33.712544 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_135_135_0
2026.05.06 08:48:33.712580 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_156_156_0
2026.05.06 08:48:33.712604 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_285_285_0
2026.05.06 08:48:33.712669 [ 32380 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_146_146_0 from disk default
2026.05.06 08:48:33.712740 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_287_287_0
2026.05.06 08:48:33.712561 [ 32379 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_156_156_0
2026.05.06 08:48:33.712786 [ 32377 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_154_154_0 from disk default
2026.05.06 08:48:33.712937 [ 32340 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_131_131_0 from disk default
2026.05.06 08:48:33.713076 [ 32369 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_135_135_0 from disk default
2026.05.06 08:48:33.713206 [ 32379 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_139_139_0 from disk default
2026.05.06 08:48:33.713251 [ 32392 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_284_284_0 from disk default
2026.05.06 08:48:33.713273 [ 32387 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_144_144_0 from disk default
2026.05.06 08:48:33.715883 [ 32376 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_155_155_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.715929 [ 32376 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_155_155_0 on disk default
2026.05.06 08:48:33.715993 [ 32376 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_155_155_0
2026.05.06 08:48:33.716121 [ 32376 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_116_116_0 from disk default
2026.05.06 08:48:33.717478 [ 32380 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_146_146_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.717537 [ 32380 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_146_146_0 on disk default
2026.05.06 08:48:33.721045 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_146_146_0
2026.05.06 08:48:33.721206 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_154_154_0 from disk default
2026.05.06 08:48:33.727703 [ 32372 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_140_140_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.727771 [ 32372 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_140_140_0 on disk default
2026.05.06 08:48:33.727845 [ 32372 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_140_140_0
2026.05.06 08:48:33.728003 [ 32372 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_115_115_0 from disk default
2026.05.06 08:48:33.728676 [ 32381 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_134_134_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.728742 [ 32381 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_134_134_0 on disk default
2026.05.06 08:48:33.728831 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_134_134_0
2026.05.06 08:48:33.728976 [ 32381 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_134_134_0 from disk default
2026.05.06 08:48:33.730406 [ 32369 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_135_135_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.730469 [ 32369 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_135_135_0 on disk default
2026.05.06 08:48:33.730566 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_135_135_0
2026.05.06 08:48:33.730735 [ 32369 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_283_283_0 from disk default
2026.05.06 08:48:33.733071 [ 32359 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_154_154_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.733121 [ 32359 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_154_154_0 on disk default
2026.05.06 08:48:33.733185 [ 32359 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_154_154_0
2026.05.06 08:48:33.733337 [ 32359 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_145_145_0 from disk default
2026.05.06 08:48:33.734769 [ 32403 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_155_155_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.734828 [ 32403 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_155_155_0 on disk default
2026.05.06 08:48:33.734891 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_155_155_0
2026.05.06 08:48:33.735021 [ 32403 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_173_173_0 from disk default
2026.05.06 08:48:33.735858 [ 32340 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_131_131_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.735912 [ 32340 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_131_131_0 on disk default
2026.05.06 08:48:33.737040 [ 32380 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_154_154_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.737087 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_154_154_0 on disk default
2026.05.06 08:48:33.737224 [ 32379 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_139_139_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.737268 [ 32379 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_139_139_0 on disk default
2026.05.06 08:48:33.737331 [ 32400 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_133_133_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.737380 [ 32400 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_133_133_0 on disk default
2026.05.06 08:48:33.738056 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_131_131_0
2026.05.06 08:48:33.738123 [ 32379 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_139_139_0
2026.05.06 08:48:33.738129 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_154_154_0
2026.05.06 08:48:33.738137 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_133_133_0
2026.05.06 08:48:33.738207 [ 32340 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_144_144_0 from disk default
2026.05.06 08:48:33.738301 [ 32394 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_143_143_0 from disk default
2026.05.06 08:48:33.738450 [ 32379 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_114_114_0 from disk default
2026.05.06 08:48:33.738933 [ 32400 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_133_133_0 from disk default
2026.05.06 08:48:33.739387 [ 32388 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_132_132_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.739436 [ 32388 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_132_132_0 on disk default
2026.05.06 08:48:33.739514 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_132_132_0
2026.05.06 08:48:33.739695 [ 32388 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_142_142_0 from disk default
2026.05.06 08:48:33.740403 [ 32340 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_144_144_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.740448 [ 32340 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_144_144_0 on disk default
2026.05.06 08:48:33.741514 [ 32392 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_284_284_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.741556 [ 32392 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_284_284_0 on disk default
2026.05.06 08:48:33.742960 [ 32394 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_143_143_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.743009 [ 32394 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_143_143_0 on disk default
2026.05.06 08:48:33.743547 [ 32382 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_156_156_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.743638 [ 32382 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_156_156_0 on disk default
2026.05.06 08:48:33.745770 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_144_144_0
2026.05.06 08:48:33.745788 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_284_284_0
2026.05.06 08:48:33.745807 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_143_143_0
2026.05.06 08:48:33.745844 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_156_156_0
2026.05.06 08:48:33.745992 [ 32383 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_138_138_0 from disk default
2026.05.06 08:48:33.746091 [ 32394 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_143_143_0 from disk default
2026.05.06 08:48:33.746264 [ 32392 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_130_130_0 from disk default
2026.05.06 08:48:33.746584 [ 32403 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_173_173_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.746631 [ 32403 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_173_173_0 on disk default
2026.05.06 08:48:33.746692 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_173_173_0
2026.05.06 08:48:33.746779 [ 32382 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_129_129_0 from disk default
2026.05.06 08:48:33.747003 [ 32403 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_132_132_0 from disk default
2026.05.06 08:48:33.749403 [ 32349 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_145_145_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.749460 [ 32349 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_145_145_0 on disk default
2026.05.06 08:48:33.750848 [ 32359 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_145_145_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.750896 [ 32359 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_145_145_0 on disk default
2026.05.06 08:48:33.751568 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_145_145_0
2026.05.06 08:48:33.751582 [ 32359 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_145_145_0
2026.05.06 08:48:33.751733 [ 32349 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_172_172_0 from disk default
2026.05.06 08:48:33.751941 [ 32359 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_171_171_0 from disk default
2026.05.06 08:48:33.752375 [ 32369 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_283_283_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.752437 [ 32369 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_283_283_0 on disk default
2026.05.06 08:48:33.752529 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_283_283_0
2026.05.06 08:48:33.752572 [ 32383 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_138_138_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.752631 [ 32383 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_138_138_0 on disk default
2026.05.06 08:48:33.752668 [ 32353 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_153_153_0 from disk default
2026.05.06 08:48:33.752704 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_138_138_0
2026.05.06 08:48:33.752823 [ 32383 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_142_142_0 from disk default
2026.05.06 08:48:33.758490 [ 32381 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_134_134_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.758526 [ 32349 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_172_172_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.758540 [ 32381 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_134_134_0 on disk default
2026.05.06 08:48:33.758571 [ 32349 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_172_172_0 on disk default
2026.05.06 08:48:33.759833 [ 32388 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_142_142_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.759881 [ 32388 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_142_142_0 on disk default
2026.05.06 08:48:33.760397 [ 32271 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 6 rows, containing 20 columns (20 merged, 0 gathered) in 0.334746359 sec., 17.924018704561924 rows/sec., 38.70 KiB/sec.
2026.05.06 08:48:33.760442 [ 32400 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_133_133_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.760512 [ 32400 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_133_133_0 on disk default
2026.05.06 08:48:33.760598 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_134_134_0
2026.05.06 08:48:33.760749 [ 32381 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_153_153_0 from disk default
2026.05.06 08:48:33.761192 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_172_172_0
2026.05.06 08:48:33.761197 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_142_142_0
2026.05.06 08:48:33.761348 [ 32349 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_137_137_0 from disk default
2026.05.06 08:48:33.761468 [ 32388 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_141_141_0 from disk default
2026.05.06 08:48:33.763213 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_133_133_0
2026.05.06 08:48:33.763360 [ 32400 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_136_136_0 from disk default
2026.05.06 08:48:33.763633 [ 32359 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_171_171_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.763707 [ 32359 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_171_171_0 on disk default
2026.05.06 08:48:33.763797 [ 32359 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_171_171_0
2026.05.06 08:48:33.763995 [ 32367 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_135_135_0 from disk default
2026.05.06 08:48:33.765632 [ 32388 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_141_141_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.765678 [ 32388 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_141_141_0 on disk default
2026.05.06 08:48:33.766874 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_141_141_0
2026.05.06 08:48:33.767023 [ 32388 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_152_152_0 from disk default
2026.05.06 08:48:33.769575 [ 32403 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_132_132_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.769631 [ 32403 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_132_132_0 on disk default
2026.05.06 08:48:33.769709 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_132_132_0
2026.05.06 08:48:33.769954 [ 32271 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Trace> MergedBlockOutputStream: filled checksums 202605_1_3_1 (state Temporary)
2026.05.06 08:48:33.769972 [ 32380 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_140_140_0 from disk default
2026.05.06 08:48:33.771728 [ 32377 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_154_154_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.771784 [ 32377 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_154_154_0 on disk default
2026.05.06 08:48:33.771854 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_154_154_0
2026.05.06 08:48:33.771989 [ 32377 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_170_170_0 from disk default
2026.05.06 08:48:33.772536 [ 32284 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 458 rows, containing 34 columns (34 merged, 0 gathered) in 0.289845113 sec., 1580.1542943385766 rows/sec., 308.21 KiB/sec.
2026.05.06 08:48:33.774071 [ 32367 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_135_135_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.774129 [ 32367 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_135_135_0 on disk default
2026.05.06 08:48:33.774201 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_135_135_0
2026.05.06 08:48:33.774359 [ 32367 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_153_153_0 from disk default
2026.05.06 08:48:33.775215 [ 32377 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_170_170_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.775264 [ 32377 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_170_170_0 on disk default
2026.05.06 08:48:33.776621 [ 32381 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_153_153_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.776653 [ 32380 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_140_140_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.776669 [ 32381 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_153_153_0 on disk default
2026.05.06 08:48:33.776700 [ 32380 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_140_140_0 on disk default
2026.05.06 08:48:33.777640 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_170_170_0
2026.05.06 08:48:33.777662 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_140_140_0
2026.05.06 08:48:33.777653 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_153_153_0
2026.05.06 08:48:33.777824 [ 32377 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_152_152_0 from disk default
2026.05.06 08:48:33.777961 [ 32381 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_141_141_0 from disk default
2026.05.06 08:48:33.778080 [ 32380 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_282_282_0 from disk default
2026.05.06 08:48:33.778657 [ 32344 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_155_155_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.778715 [ 32344 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_155_155_0 on disk default
2026.05.06 08:48:33.778790 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_155_155_0
2026.05.06 08:48:33.778835 [ 32392 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_130_130_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.778879 [ 32392 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_130_130_0 on disk default
2026.05.06 08:48:33.778933 [ 32344 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_134_134_0 from disk default
2026.05.06 08:48:33.778941 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_130_130_0
2026.05.06 08:48:33.779039 [ 32392 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_281_281_0 from disk default
2026.05.06 08:48:33.782234 [ 32349 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_137_137_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.782283 [ 32349 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_137_137_0 on disk default
2026.05.06 08:48:33.782433 [ 32400 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_136_136_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.782475 [ 32400 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_136_136_0 on disk default
2026.05.06 08:48:33.782548 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_136_136_0
2026.05.06 08:48:33.782586 [ 32382 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_129_129_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.782640 [ 32382 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_129_129_0 on disk default
2026.05.06 08:48:33.782671 [ 32400 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_151_151_0 from disk default
2026.05.06 08:48:33.782713 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_129_129_0
2026.05.06 08:48:33.782830 [ 32382 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_140_140_0 from disk default
2026.05.06 08:48:33.784576 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_137_137_0
2026.05.06 08:48:33.784767 [ 32349 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_152_152_0 from disk default
2026.05.06 08:48:33.786292 [ 32367 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_153_153_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.786341 [ 32367 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_153_153_0 on disk default
2026.05.06 08:48:33.786404 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_153_153_0
2026.05.06 08:48:33.786533 [ 32367 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_131_131_0 from disk default
2026.05.06 08:48:33.791934 [ 32377 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_152_152_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.791985 [ 32377 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_152_152_0 on disk default
2026.05.06 08:48:33.792051 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_152_152_0
2026.05.06 08:48:33.792210 [ 32338 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_133_133_0 from disk default
2026.05.06 08:48:33.794227 [ 32367 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_131_131_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.794272 [ 32380 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_282_282_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.794277 [ 32367 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_131_131_0 on disk default
2026.05.06 08:48:33.794333 [ 32380 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_282_282_0 on disk default
2026.05.06 08:48:33.794380 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_131_131_0
2026.05.06 08:48:33.794399 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_282_282_0
2026.05.06 08:48:33.794516 [ 32367 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_151_151_0 from disk default
2026.05.06 08:48:33.794741 [ 32380 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_280_280_0 from disk default
2026.05.06 08:48:33.795859 [ 32344 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_134_134_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.795901 [ 32344 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_134_134_0 on disk default
2026.05.06 08:48:33.796040 [ 32338 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_133_133_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.796105 [ 32338 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_133_133_0 on disk default
2026.05.06 08:48:33.796176 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_133_133_0
2026.05.06 08:48:33.796317 [ 32338 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_139_139_0 from disk default
2026.05.06 08:48:33.796462 [ 32285 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 758 rows, containing 15 columns (15 merged, 0 gathered) in 0.310749979 sec., 2439.2600200304437 rows/sec., 929.50 KiB/sec.
2026.05.06 08:48:33.797962 [ 32353 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_153_153_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.798010 [ 32353 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_153_153_0 on disk default
2026.05.06 08:48:33.798075 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_153_153_0
2026.05.06 08:48:33.798244 [ 32353 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_169_169_0 from disk default
2026.05.06 08:48:33.798793 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_134_134_0
2026.05.06 08:48:33.798949 [ 32344 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_139_139_0 from disk default
2026.05.06 08:48:33.803400 [ 32284 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Trace> MergedBlockOutputStream: filled checksums 202605_167_170_1 (state Temporary)
2026.05.06 08:48:33.804377 [ 32353 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_169_169_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.804426 [ 32353 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_169_169_0 on disk default
2026.05.06 08:48:33.804502 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_169_169_0
2026.05.06 08:48:33.804628 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 2085 rows, containing 19 columns (19 merged, 0 gathered) in 0.364480626 sec., 5720.468664910602 rows/sec., 1.10 MiB/sec.
2026.05.06 08:48:33.804716 [ 32395 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_138_138_0 from disk default
2026.05.06 08:48:33.805115 [ 32349 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_152_152_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.805166 [ 32349 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_152_152_0 on disk default
2026.05.06 08:48:33.805228 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_152_152_0
2026.05.06 08:48:33.806589 [ 32344 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_139_139_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.806659 [ 32344 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_139_139_0 on disk default
2026.05.06 08:48:33.806737 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_139_139_0
2026.05.06 08:48:33.806941 [ 32344 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_128_128_0 from disk default
2026.05.06 08:48:33.807112 [ 32349 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_150_150_0 from disk default
2026.05.06 08:48:33.808041 [ 32395 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_138_138_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.808087 [ 32395 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_138_138_0 on disk default
2026.05.06 08:48:33.808137 [ 32367 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_151_151_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.808183 [ 32367 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_151_151_0 on disk default
2026.05.06 08:48:33.809833 [ 32392 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_281_281_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.809898 [ 32392 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_281_281_0 on disk default
2026.05.06 08:48:33.810382 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_138_138_0
2026.05.06 08:48:33.810559 [ 32395 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_149_149_0 from disk default
2026.05.06 08:48:33.810573 [ 32285 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Trace> MergedBlockOutputStream: filled checksums 202605_171_175_1 (state Temporary)
2026.05.06 08:48:33.810785 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_151_151_0
2026.05.06 08:48:33.810806 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_281_281_0
2026.05.06 08:48:33.810945 [ 32367 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_127_127_0 from disk default
2026.05.06 08:48:33.811724 [ 32392 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_138_138_0 from disk default
2026.05.06 08:48:33.815180 [ 32380 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_280_280_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.815250 [ 32380 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_280_280_0 on disk default
2026.05.06 08:48:33.815339 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_280_280_0
2026.05.06 08:48:33.815514 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_151_151_0 from disk default
2026.05.06 08:48:33.816462 [ 32394 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_143_143_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.816537 [ 32394 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_143_143_0 on disk default
2026.05.06 08:48:33.816747 [ 32387 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_144_144_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.817329 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_143_143_0
2026.05.06 08:48:33.817762 [ 31633 ] {} <Debug> SystemLog: Creating system.part_log from part_log
2026.05.06 08:48:33.818043 [ 32401 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_132_132_0 from disk default
2026.05.06 08:48:33.819464 [ 32387 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_144_144_0 on disk default
2026.05.06 08:48:33.819553 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_144_144_0
2026.05.06 08:48:33.819764 [ 32399 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_131_131_0 from disk default
2026.05.06 08:48:33.821029 [ 32395 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_149_149_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.821080 [ 32395 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_149_149_0 on disk default
2026.05.06 08:48:33.821146 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_149_149_0
2026.05.06 08:48:33.821146 [ 32380 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_151_151_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.821243 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_151_151_0 on disk default
2026.05.06 08:48:33.821316 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_151_151_0
2026.05.06 08:48:33.821369 [ 32339 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_130_130_0 from disk default
2026.05.06 08:48:33.821441 [ 32380 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_137_137_0 from disk default
2026.05.06 08:48:33.824193 [ 32349 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_150_150_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.824248 [ 32349 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_150_150_0 on disk default
2026.05.06 08:48:33.824335 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_150_150_0
2026.05.06 08:48:33.824531 [ 32349 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_168_168_0 from disk default
2026.05.06 08:48:33.825558 [ 32339 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_130_130_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.825606 [ 32339 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_130_130_0 on disk default
2026.05.06 08:48:33.825676 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_130_130_0
2026.05.06 08:48:33.825783 [ 32339 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_150_150_0 from disk default
2026.05.06 08:48:33.826041 [ 32401 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_132_132_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.826106 [ 32401 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_132_132_0 on disk default
2026.05.06 08:48:33.826189 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_132_132_0
2026.05.06 08:48:33.826344 [ 32401 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_130_130_0 from disk default
2026.05.06 08:48:33.833853 [ 32383 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_142_142_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.833909 [ 32383 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_142_142_0 on disk default
2026.05.06 08:48:33.833981 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_142_142_0
2026.05.06 08:48:33.834121 [ 32383 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_149_149_0 from disk default
2026.05.06 08:48:33.835112 [ 32399 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_131_131_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.835176 [ 32399 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_131_131_0 on disk default
2026.05.06 08:48:33.835264 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_131_131_0
2026.05.06 08:48:33.835396 [ 32399 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_137_137_0 from disk default
2026.05.06 08:48:33.837929 [ 32349 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_168_168_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.837980 [ 32349 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_168_168_0 on disk default
2026.05.06 08:48:33.838061 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_168_168_0
2026.05.06 08:48:33.838257 [ 32378 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_136_136_0 from disk default
2026.05.06 08:48:33.840715 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Trace> MergedBlockOutputStream: filled checksums 202605_151_154_1 (state Temporary)
2026.05.06 08:48:33.844690 [ 32401 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_130_130_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.844757 [ 32401 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_130_130_0 on disk default
2026.05.06 08:48:33.844853 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_130_130_0
2026.05.06 08:48:33.845095 [ 32337 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_135_135_0 from disk default
2026.05.06 08:48:33.846113 [ 32273 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Trace> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): Renaming temporary part tmp_merge_202605_1_3_1 to 202605_1_3_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:33.846219 [ 32399 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_137_137_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.846284 [ 32399 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_137_137_0 on disk default
2026.05.06 08:48:33.846364 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_137_137_0
2026.05.06 08:48:33.846416 [ 32273 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Test> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee): preparePartForCommit: inserting 202605_1_3_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:33.846541 [ 32399 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_134_134_0 from disk default
2026.05.06 08:48:33.846651 [ 32344 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_128_128_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.846702 [ 32344 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_128_128_0 on disk default
2026.05.06 08:48:33.846771 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_128_128_0
2026.05.06 08:48:33.846871 [ 32339 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_150_150_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.846925 [ 32344 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_150_150_0 from disk default
2026.05.06 08:48:33.846944 [ 32339 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_150_150_0 on disk default
2026.05.06 08:48:33.847032 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_150_150_0
2026.05.06 08:48:33.847274 [ 32403 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_148_148_0 from disk default
2026.05.06 08:48:33.848212 [ 32337 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_135_135_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.848263 [ 32337 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_135_135_0 on disk default
2026.05.06 08:48:33.848299 [ 32378 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_136_136_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.848345 [ 32378 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_136_136_0 on disk default
2026.05.06 08:48:33.848449 [ 32273 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Trace> system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee) (MergerMutator): Merged 3 parts: [202605_1_1_0, 202605_3_3_0] -> 202605_1_3_1
2026.05.06 08:48:33.849247 [ 32337 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_135_135_0
2026.05.06 08:48:33.849249 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_136_136_0
2026.05.06 08:48:33.849429 [ 32337 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_148_148_0 from disk default
2026.05.06 08:48:33.849620 [ 32378 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_129_129_0 from disk default
2026.05.06 08:48:33.850420 [ 32273 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee), part: 202605_1_1_0
2026.05.06 08:48:33.850575 [ 32273 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee), part: 202605_2_2_0
2026.05.06 08:48:33.850651 [ 32273 ] {3a4912e0-e69b-4b16-862e-ada94b4c32ee::202605_1_3_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.backup_log (3a4912e0-e69b-4b16-862e-ada94b4c32ee), part: 202605_3_3_0
2026.05.06 08:48:33.851378 [ 32399 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_134_134_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.851419 [ 32399 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_134_134_0 on disk default
2026.05.06 08:48:33.851493 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_134_134_0
2026.05.06 08:48:33.851628 [ 32399 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_147_147_0 from disk default
2026.05.06 08:48:33.853641 [ 32383 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_149_149_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.853697 [ 32383 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_149_149_0 on disk default
2026.05.06 08:48:33.853759 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_149_149_0
2026.05.06 08:48:33.853899 [ 32383 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_167_167_0 from disk default
2026.05.06 08:48:33.855152 [ 32403 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_148_148_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.855198 [ 32403 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_148_148_0 on disk default
2026.05.06 08:48:33.856162 [ 32400 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_151_151_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.856211 [ 32400 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_151_151_0 on disk default
2026.05.06 08:48:33.856631 [ 32388 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_152_152_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.856699 [ 32388 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_152_152_0 on disk default
2026.05.06 08:48:33.858037 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_148_148_0
2026.05.06 08:48:33.858207 [ 32403 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_147_147_0 from disk default
2026.05.06 08:48:33.858305 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_151_151_0
2026.05.06 08:48:33.858335 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_152_152_0
2026.05.06 08:48:33.858441 [ 32400 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_113_113_0 from disk default
2026.05.06 08:48:33.858635 [ 32388 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_136_136_0 from disk default
2026.05.06 08:48:33.861123 [ 32337 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_148_148_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.861176 [ 32337 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_148_148_0 on disk default
2026.05.06 08:48:33.861239 [ 32337 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_148_148_0
2026.05.06 08:48:33.861387 [ 32337 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_129_129_0 from disk default
2026.05.06 08:48:33.863763 [ 32383 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_167_167_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.863811 [ 32383 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_167_167_0 on disk default
2026.05.06 08:48:33.863872 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_167_167_0
2026.05.06 08:48:33.864021 [ 32383 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_279_279_0 from disk default
2026.05.06 08:48:33.865445 [ 32378 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_129_129_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.865519 [ 32378 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_129_129_0 on disk default
2026.05.06 08:48:33.865989 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_129_129_0
2026.05.06 08:48:33.866132 [ 32378 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_128_128_0 from disk default
2026.05.06 08:48:33.869547 [ 32367 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_127_127_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.869642 [ 32367 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_127_127_0 on disk default
2026.05.06 08:48:33.869754 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_127_127_0
2026.05.06 08:48:33.869979 [ 32367 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_146_146_0 from disk default
2026.05.06 08:48:33.875317 [ 32403 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_147_147_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.875380 [ 32403 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_147_147_0 on disk default
2026.05.06 08:48:33.875449 [ 32381 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_141_141_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.875461 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_147_147_0
2026.05.06 08:48:33.875509 [ 32381 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_141_141_0 on disk default
2026.05.06 08:48:33.875579 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_141_141_0
2026.05.06 08:48:33.875653 [ 32403 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_127_127_0 from disk default
2026.05.06 08:48:33.875713 [ 32381 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_133_133_0 from disk default
2026.05.06 08:48:33.877593 [ 32399 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_147_147_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.877639 [ 32399 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_147_147_0 on disk default
2026.05.06 08:48:33.877700 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_147_147_0
2026.05.06 08:48:33.877843 [ 32399 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_126_126_0 from disk default
2026.05.06 08:48:33.878626 [ 32382 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_140_140_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.878683 [ 32382 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_140_140_0 on disk default
2026.05.06 08:48:33.879229 [ 32378 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_128_128_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.879283 [ 32378 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_128_128_0 on disk default
2026.05.06 08:48:33.879470 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_128_128_0
2026.05.06 08:48:33.879653 [ 32378 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_166_166_0 from disk default
2026.05.06 08:48:33.882282 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_140_140_0
2026.05.06 08:48:33.882452 [ 32382 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_125_125_0 from disk default
2026.05.06 08:48:33.883285 [ 32381 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_133_133_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.883333 [ 32381 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_133_133_0 on disk default
2026.05.06 08:48:33.883395 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_133_133_0
2026.05.06 08:48:33.883842 [ 32337 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_129_129_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.883894 [ 32337 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_129_129_0 on disk default
2026.05.06 08:48:33.883939 [ 32338 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_139_139_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.883957 [ 32337 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_129_129_0
2026.05.06 08:48:33.883995 [ 32338 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_139_139_0 on disk default
2026.05.06 08:48:33.884077 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_139_139_0
2026.05.06 08:48:33.885562 [ 32353 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_149_149_0 from disk default
2026.05.06 08:48:33.885837 [ 32339 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_165_165_0 from disk default
2026.05.06 08:48:33.885952 [ 32381 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_145_145_0 from disk default
2026.05.06 08:48:33.888330 [ 32339 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_165_165_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.888380 [ 32339 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_165_165_0 on disk default
2026.05.06 08:48:33.888879 [ 32378 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_166_166_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.888928 [ 32378 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_166_166_0 on disk default
2026.05.06 08:48:33.889083 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_166_166_0
2026.05.06 08:48:33.889089 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_165_165_0
2026.05.06 08:48:33.889245 [ 32378 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_128_128_0 from disk default
2026.05.06 08:48:33.889316 [ 32339 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_127_127_0 from disk default
2026.05.06 08:48:33.890255 [ 32399 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_126_126_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.890312 [ 32399 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_126_126_0 on disk default
2026.05.06 08:48:33.890382 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_126_126_0
2026.05.06 08:48:33.890555 [ 32399 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_126_126_0 from disk default
2026.05.06 08:48:33.890808 [ 32367 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_146_146_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.890922 [ 32367 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_146_146_0 on disk default
2026.05.06 08:48:33.892330 [ 32285 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Renaming temporary part tmp_merge_202605_171_175_1 to 202605_171_175_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:33.892787 [ 32382 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_125_125_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.892832 [ 32382 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_125_125_0 on disk default
2026.05.06 08:48:33.892897 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_125_125_0
2026.05.06 08:48:33.893022 [ 32382 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_132_132_0 from disk default
2026.05.06 08:48:33.893829 [ 32383 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_279_279_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.893876 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_146_146_0
2026.05.06 08:48:33.893903 [ 32383 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_279_279_0 on disk default
2026.05.06 08:48:33.894062 [ 32367 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_126_126_0 from disk default
2026.05.06 08:48:33.895034 [ 32403 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_127_127_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.895080 [ 32403 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_127_127_0 on disk default
2026.05.06 08:48:33.895550 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_279_279_0
2026.05.06 08:48:33.895570 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_127_127_0
2026.05.06 08:48:33.895727 [ 32383 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_112_112_0 from disk default
2026.05.06 08:48:33.896591 [ 32339 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_127_127_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.896648 [ 32339 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_127_127_0 on disk default
2026.05.06 08:48:33.896717 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_127_127_0
2026.05.06 08:48:33.896994 [ 32387 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_125_125_0 from disk default
2026.05.06 08:48:33.897751 [ 32339 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_135_135_0 from disk default
2026.05.06 08:48:33.898322 [ 32273 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 6.15 MiB.
2026.05.06 08:48:33.898658 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Renaming temporary part tmp_merge_202605_151_154_1 to 202605_151_154_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:33.899648 [ 32382 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_132_132_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.899712 [ 32382 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_132_132_0 on disk default
2026.05.06 08:48:33.901264 [ 32285 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): preparePartForCommit: inserting 202605_171_175_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:33.901339 [ 32285 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa) (MergerMutator): Merged 5 parts: [202605_171_171_0, 202605_175_175_0] -> 202605_171_175_1
2026.05.06 08:48:33.901264 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): preparePartForCommit: inserting 202605_151_154_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:33.902507 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383) (MergerMutator): Merged 4 parts: [202605_151_151_0, 202605_154_154_0] -> 202605_151_154_1
2026.05.06 08:48:33.902522 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_132_132_0
2026.05.06 08:48:33.902554 [ 32378 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_128_128_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.902611 [ 32378 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_128_128_0 on disk default
2026.05.06 08:48:33.902624 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_151_151_0
2026.05.06 08:48:33.902622 [ 32285 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_171_171_0
2026.05.06 08:48:33.902689 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_128_128_0
2026.05.06 08:48:33.902695 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_152_152_0
2026.05.06 08:48:33.902705 [ 32285 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_172_172_0
2026.05.06 08:48:33.902748 [ 32373 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_134_134_0 from disk default
2026.05.06 08:48:33.902769 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_153_153_0
2026.05.06 08:48:33.902807 [ 32285 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_173_173_0
2026.05.06 08:48:33.902842 [ 32386 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_144_144_0 from disk default
2026.05.06 08:48:33.902855 [ 32277 ] {41d85242-310c-4d42-9b59-fea8708c8383::202605_151_154_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_154_154_0
2026.05.06 08:48:33.902864 [ 32285 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_174_174_0
2026.05.06 08:48:33.902937 [ 32285 ] {aa0461d4-91df-4949-8bb0-34b45a966ffa::202605_171_175_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_175_175_0
2026.05.06 08:48:33.903468 [ 32399 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_126_126_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.903526 [ 32399 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_126_126_0 on disk default
2026.05.06 08:48:33.903587 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_126_126_0
2026.05.06 08:48:33.903768 [ 32399 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_111_111_0 from disk default
2026.05.06 08:48:33.905420 [ 32344 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_150_150_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.905468 [ 32344 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_150_150_0 on disk default
2026.05.06 08:48:33.905539 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_150_150_0
2026.05.06 08:48:33.906928 [ 32344 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_148_148_0 from disk default
2026.05.06 08:48:33.907853 [ 32381 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_145_145_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.907958 [ 32381 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_145_145_0 on disk default
2026.05.06 08:48:33.910691 [ 32380 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_137_137_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.910774 [ 32380 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_137_137_0 on disk default
2026.05.06 08:48:33.910873 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_137_137_0
2026.05.06 08:48:33.911060 [ 32380 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_278_278_0 from disk default
2026.05.06 08:48:33.911396 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_145_145_0
2026.05.06 08:48:33.911475 [ 32284 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Renaming temporary part tmp_merge_202605_167_170_1 to 202605_167_170_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:33.911714 [ 32284 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): preparePartForCommit: inserting 202605_167_170_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:33.911726 [ 32381 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_131_131_0 from disk default
2026.05.06 08:48:33.911767 [ 32284 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185) (MergerMutator): Merged 4 parts: [202605_167_167_0, 202605_170_170_0] -> 202605_167_170_1
2026.05.06 08:48:33.911856 [ 32284 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_167_167_0
2026.05.06 08:48:33.911908 [ 32284 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_168_168_0
2026.05.06 08:48:33.911953 [ 32284 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_169_169_0
2026.05.06 08:48:33.911997 [ 32284 ] {f32b09c6-f6ce-4ba9-8f8a-21554928d185::202605_167_170_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_170_170_0
2026.05.06 08:48:33.913925 [ 32386 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_144_144_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.913980 [ 32386 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_144_144_0 on disk default
2026.05.06 08:48:33.914045 [ 32386 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_144_144_0
2026.05.06 08:48:33.914200 [ 32386 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_124_124_0 from disk default
2026.05.06 08:48:33.917342 [ 32353 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_149_149_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.917411 [ 32353 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_149_149_0 on disk default
2026.05.06 08:48:33.917505 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_149_149_0
2026.05.06 08:48:33.917668 [ 32353 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_277_277_0 from disk default
2026.05.06 08:48:33.918743 [ 32381 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_131_131_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.918798 [ 32381 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_131_131_0 on disk default
2026.05.06 08:48:33.928012 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_131_131_0
2026.05.06 08:48:33.928209 [ 32277 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 6.15 MiB.
2026.05.06 08:48:33.928253 [ 32359 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_276_276_0 from disk default
2026.05.06 08:48:33.928654 [ 32285 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 6.15 MiB.
2026.05.06 08:48:33.929152 [ 32392 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_138_138_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.929215 [ 32392 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_138_138_0 on disk default
2026.05.06 08:48:33.929296 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_138_138_0
2026.05.06 08:48:33.929522 [ 32392 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_164_164_0 from disk default
2026.05.06 08:48:33.930508 [ 32387 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_125_125_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.930557 [ 32387 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_125_125_0 on disk default
2026.05.06 08:48:33.930624 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_125_125_0
2026.05.06 08:48:33.930782 [ 32387 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_130_130_0 from disk default
2026.05.06 08:48:33.933710 [ 32392 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_164_164_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.933761 [ 32392 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_164_164_0 on disk default
2026.05.06 08:48:33.933826 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_164_164_0
2026.05.06 08:48:33.934004 [ 32374 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_133_133_0 from disk default
2026.05.06 08:48:33.936536 [ 32284 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 6.16 MiB.
2026.05.06 08:48:33.937106 [ 32367 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_126_126_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.937172 [ 32367 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_126_126_0 on disk default
2026.05.06 08:48:33.937217 [ 32387 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_130_130_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.937276 [ 32387 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_130_130_0 on disk default
2026.05.06 08:48:33.938855 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_130_130_0
2026.05.06 08:48:33.938886 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_126_126_0
2026.05.06 08:48:33.939021 [ 32387 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_146_146_0 from disk default
2026.05.06 08:48:33.939243 [ 32367 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_143_143_0 from disk default
2026.05.06 08:48:33.941682 [ 32359 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_276_276_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.941734 [ 32359 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_276_276_0 on disk default
2026.05.06 08:48:33.941798 [ 32359 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_276_276_0
2026.05.06 08:48:33.941957 [ 32340 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_142_142_0 from disk default
2026.05.06 08:48:33.943558 [ 32380 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_278_278_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.943624 [ 32380 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_278_278_0 on disk default
2026.05.06 08:48:33.943702 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_278_278_0
2026.05.06 08:48:33.943855 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_145_145_0 from disk default
2026.05.06 08:48:33.944063 [ 32386 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_124_124_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.944187 [ 32386 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_124_124_0 on disk default
2026.05.06 08:48:33.945991 [ 32386 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_124_124_0
2026.05.06 08:48:33.946181 [ 32394 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_144_144_0 from disk default
2026.05.06 08:48:33.947580 [ 32353 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_277_277_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.947652 [ 32353 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_277_277_0 on disk default
2026.05.06 08:48:33.949163 [ 32340 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_142_142_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.949223 [ 32340 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_142_142_0 on disk default
2026.05.06 08:48:33.949284 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_142_142_0
2026.05.06 08:48:33.949383 [ 32367 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_143_143_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.949409 [ 32340 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_124_124_0 from disk default
2026.05.06 08:48:33.949428 [ 32367 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_143_143_0 on disk default
2026.05.06 08:48:33.949507 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_143_143_0
2026.05.06 08:48:33.949621 [ 32367 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_141_141_0 from disk default
2026.05.06 08:48:33.950116 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_277_277_0
2026.05.06 08:48:33.951422 [ 32353 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_143_143_0 from disk default
2026.05.06 08:48:33.954146 [ 32388 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_136_136_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.954208 [ 32388 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_136_136_0 on disk default
2026.05.06 08:48:33.954292 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_136_136_0
2026.05.06 08:48:33.954402 [ 32387 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_146_146_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.954462 [ 32387 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_146_146_0 on disk default
2026.05.06 08:48:33.954490 [ 32388 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_142_142_0 from disk default
2026.05.06 08:48:33.954539 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_146_146_0
2026.05.06 08:48:33.954599 [ 32394 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_144_144_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.954652 [ 32394 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_144_144_0 on disk default
2026.05.06 08:48:33.954660 [ 32387 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_147_147_0 from disk default
2026.05.06 08:48:33.956624 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_144_144_0
2026.05.06 08:48:33.956808 [ 32394 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_132_132_0 from disk default
2026.05.06 08:48:33.959752 [ 32340 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_124_124_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.959801 [ 32340 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_124_124_0 on disk default
2026.05.06 08:48:33.959863 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_124_124_0
2026.05.06 08:48:33.959879 [ 32380 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_145_145_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.959933 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_145_145_0 on disk default
2026.05.06 08:48:33.960010 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_145_145_0
2026.05.06 08:48:33.960120 [ 32380 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_146_146_0 from disk default
2026.05.06 08:48:33.960160 [ 32340 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_140_140_0 from disk default
2026.05.06 08:48:33.964350 [ 32367 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_141_141_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.964399 [ 32344 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_148_148_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.964412 [ 32367 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_141_141_0 on disk default
2026.05.06 08:48:33.964458 [ 32344 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_148_148_0 on disk default
2026.05.06 08:48:33.964508 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_141_141_0
2026.05.06 08:48:33.964546 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_148_148_0
2026.05.06 08:48:33.964673 [ 32349 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_129_129_0 from disk default
2026.05.06 08:48:33.964798 [ 32344 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_125_125_0 from disk default
2026.05.06 08:48:33.968431 [ 32388 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_142_142_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.968505 [ 32388 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_142_142_0 on disk default
2026.05.06 08:48:33.968587 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_142_142_0
2026.05.06 08:48:33.968761 [ 32388 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_110_110_0 from disk default
2026.05.06 08:48:33.968922 [ 32353 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_143_143_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.968967 [ 32353 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_143_143_0 on disk default
2026.05.06 08:48:33.968999 [ 32373 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_134_134_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.969041 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_143_143_0
2026.05.06 08:48:33.969047 [ 32373 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_134_134_0 on disk default
2026.05.06 08:48:33.969109 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_134_134_0
2026.05.06 08:48:33.969163 [ 32339 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_135_135_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.969170 [ 32353 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_163_163_0 from disk default
2026.05.06 08:48:33.969219 [ 32339 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_135_135_0 on disk default
2026.05.06 08:48:33.969286 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_135_135_0
2026.05.06 08:48:33.969368 [ 32373 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_131_131_0 from disk default
2026.05.06 08:48:33.969541 [ 32339 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_123_123_0 from disk default
2026.05.06 08:48:33.969551 [ 32349 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_129_129_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.969640 [ 32349 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_129_129_0 on disk default
2026.05.06 08:48:33.970113 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_129_129_0
2026.05.06 08:48:33.970291 [ 32349 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_139_139_0 from disk default
2026.05.06 08:48:33.971459 [ 32353 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_163_163_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.971519 [ 32353 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_163_163_0 on disk default
2026.05.06 08:48:33.972592 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_163_163_0
2026.05.06 08:48:33.973161 [ 32397 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_109_109_0 from disk default
2026.05.06 08:48:33.976825 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 7058 rows, containing 14 columns (14 merged, 0 gathered) in 0.547828495 sec., 12883.594162074392 rows/sec., 2.10 MiB/sec.
2026.05.06 08:48:33.976932 [ 32349 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_139_139_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.976994 [ 32349 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_139_139_0 on disk default
2026.05.06 08:48:33.977081 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_139_139_0
2026.05.06 08:48:33.977279 [ 32359 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_108_108_0 from disk default
2026.05.06 08:48:33.985023 [ 32340 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_140_140_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.985085 [ 32340 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_140_140_0 on disk default
2026.05.06 08:48:33.985170 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_140_140_0
2026.05.06 08:48:33.985391 [ 32339 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_123_123_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.985398 [ 32371 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_128_128_0 from disk default
2026.05.06 08:48:33.985453 [ 32339 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_123_123_0 on disk default
2026.05.06 08:48:33.985543 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_123_123_0
2026.05.06 08:48:33.985690 [ 32386 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_107_107_0 from disk default
2026.05.06 08:48:33.987964 [ 32371 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_128_128_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.988014 [ 32371 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_128_128_0 on disk default
2026.05.06 08:48:33.988074 [ 32371 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_128_128_0
2026.05.06 08:48:33.988224 [ 32371 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_124_124_0 from disk default
2026.05.06 08:48:33.989892 [ 32374 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_133_133_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.989953 [ 32374 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_133_133_0 on disk default
2026.05.06 08:48:33.990034 [ 32374 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_133_133_0
2026.05.06 08:48:33.990201 [ 32374 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_138_138_0 from disk default
2026.05.06 08:48:33.990251 [ 32344 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_125_125_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.990312 [ 32344 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_125_125_0 on disk default
2026.05.06 08:48:33.990390 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_125_125_0
2026.05.06 08:48:33.990541 [ 32344 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_123_123_0 from disk default
2026.05.06 08:48:33.999173 [ 32371 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_124_124_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:33.999224 [ 32371 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_124_124_0 on disk default
2026.05.06 08:48:33.999290 [ 32371 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_124_124_0
2026.05.06 08:48:33.999448 [ 32371 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_145_145_0 from disk default
2026.05.06 08:48:34.001699 [ 31633 ] {} <Debug> SystemLog: Creating system.trace_log from trace_log
2026.05.06 08:48:34.002928 [ 32344 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_123_123_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.002974 [ 32344 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_123_123_0 on disk default
2026.05.06 08:48:34.003636 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_123_123_0
2026.05.06 08:48:34.003792 [ 32344 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_122_122_0 from disk default
2026.05.06 08:48:34.005041 [ 32374 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_138_138_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.005105 [ 32374 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_138_138_0 on disk default
2026.05.06 08:48:34.005188 [ 32374 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_138_138_0
2026.05.06 08:48:34.005375 [ 32377 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_144_144_0 from disk default
2026.05.06 08:48:34.012469 [ 32394 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_132_132_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.012536 [ 32394 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_132_132_0 on disk default
2026.05.06 08:48:34.012601 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_132_132_0
2026.05.06 08:48:34.012732 [ 32394 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_141_141_0 from disk default
2026.05.06 08:48:34.013086 [ 32344 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_122_122_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.013132 [ 32344 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_122_122_0 on disk default
2026.05.06 08:48:34.013193 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_122_122_0
2026.05.06 08:48:34.013321 [ 32344 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_122_122_0 from disk default
2026.05.06 08:48:34.023049 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Trace> MergedBlockOutputStream: filled checksums 202605_1_175_35 (state Temporary)
2026.05.06 08:48:34.024650 [ 32394 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_141_141_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.024713 [ 32394 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_141_141_0 on disk default
2026.05.06 08:48:34.024886 [ 32387 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_147_147_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.024935 [ 32387 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_147_147_0 on disk default
2026.05.06 08:48:34.028397 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_147_147_0
2026.05.06 08:48:34.028620 [ 32387 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_275_275_0 from disk default
2026.05.06 08:48:34.030611 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_141_141_0
2026.05.06 08:48:34.030627 [ 32344 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_122_122_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.030749 [ 32344 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_122_122_0 on disk default
2026.05.06 08:48:34.030843 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_122_122_0
2026.05.06 08:48:34.030851 [ 32394 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_140_140_0 from disk default
2026.05.06 08:48:34.030989 [ 32344 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_121_121_0 from disk default
2026.05.06 08:48:34.032438 [ 32380 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_146_146_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.032504 [ 32380 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_146_146_0 on disk default
2026.05.06 08:48:34.032570 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_146_146_0
2026.05.06 08:48:34.032707 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_139_139_0 from disk default
2026.05.06 08:48:34.034030 [ 32377 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_144_144_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.034055 [ 32373 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_131_131_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.034072 [ 32377 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_144_144_0 on disk default
2026.05.06 08:48:34.034119 [ 32373 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_131_131_0 on disk default
2026.05.06 08:48:34.034152 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_144_144_0
2026.05.06 08:48:34.034214 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_131_131_0
2026.05.06 08:48:34.034405 [ 32392 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_143_143_0 from disk default
2026.05.06 08:48:34.034459 [ 32338 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_130_130_0 from disk default
2026.05.06 08:48:34.043145 [ 32394 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_140_140_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.043200 [ 32394 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_140_140_0 on disk default
2026.05.06 08:48:34.043268 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_140_140_0
2026.05.06 08:48:34.044128 [ 32394 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_162_162_0 from disk default
2026.05.06 08:48:34.046062 [ 32344 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_121_121_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.046126 [ 32344 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_121_121_0 on disk default
2026.05.06 08:48:34.046207 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_121_121_0
2026.05.06 08:48:34.046384 [ 32387 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_275_275_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.046441 [ 32387 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_275_275_0 on disk default
2026.05.06 08:48:34.046524 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_275_275_0
2026.05.06 08:48:34.046569 [ 32353 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_161_161_0 from disk default
2026.05.06 08:48:34.046669 [ 32387 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_160_160_0 from disk default
2026.05.06 08:48:34.046945 [ 32380 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_139_139_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.047003 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_139_139_0 on disk default
2026.05.06 08:48:34.047086 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_139_139_0
2026.05.06 08:48:34.047294 [ 32339 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_159_159_0 from disk default
2026.05.06 08:48:34.049342 [ 32339 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_159_159_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.049401 [ 32339 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_159_159_0 on disk default
2026.05.06 08:48:34.049540 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_159_159_0
2026.05.06 08:48:34.049736 [ 32340 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_121_121_0 from disk default
2026.05.06 08:48:34.050736 [ 32353 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_161_161_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.050789 [ 32353 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_161_161_0 on disk default
2026.05.06 08:48:34.050859 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_161_161_0
2026.05.06 08:48:34.050986 [ 32353 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_127_127_0 from disk default
2026.05.06 08:48:34.054334 [ 32353 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_127_127_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.054386 [ 32353 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_127_127_0 on disk default
2026.05.06 08:48:34.054452 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_127_127_0
2026.05.06 08:48:34.054597 [ 32353 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_120_120_0 from disk default
2026.05.06 08:48:34.054634 [ 32340 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_121_121_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.054685 [ 32340 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_121_121_0 on disk default
2026.05.06 08:48:34.054748 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_121_121_0
2026.05.06 08:48:34.054903 [ 32340 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_126_126_0 from disk default
2026.05.06 08:48:34.057628 [ 32371 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_145_145_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.057679 [ 32371 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_145_145_0 on disk default
2026.05.06 08:48:34.057742 [ 32371 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_145_145_0
2026.05.06 08:48:34.057887 [ 32371 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_106_106_0 from disk default
2026.05.06 08:48:34.058141 [ 32392 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_143_143_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.058191 [ 32392 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_143_143_0 on disk default
2026.05.06 08:48:34.059491 [ 32387 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_160_160_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.059555 [ 32387 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_160_160_0 on disk default
2026.05.06 08:48:34.059656 [ 32340 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_126_126_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.059719 [ 32340 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_126_126_0 on disk default
2026.05.06 08:48:34.061070 [ 32394 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_162_162_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.061114 [ 32394 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_162_162_0 on disk default
2026.05.06 08:48:34.061318 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_143_143_0
2026.05.06 08:48:34.061336 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_160_160_0
2026.05.06 08:48:34.061350 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_126_126_0
2026.05.06 08:48:34.061360 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_162_162_0
2026.05.06 08:48:34.061547 [ 32387 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_137_137_0 from disk default
2026.05.06 08:48:34.061610 [ 32374 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_120_120_0 from disk default
2026.05.06 08:48:34.061658 [ 32342 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_119_119_0 from disk default
2026.05.06 08:48:34.061768 [ 32394 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_142_142_0 from disk default
2026.05.06 08:48:34.066270 [ 32361 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_131_131_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.066325 [ 32361 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_131_131_0 on disk default
2026.05.06 08:48:34.066392 [ 32361 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_131_131_0
2026.05.06 08:48:34.066535 [ 32361 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_125_125_0 from disk default
2026.05.06 08:48:34.067411 [ 32353 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_120_120_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.067475 [ 32353 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_120_120_0 on disk default
2026.05.06 08:48:34.068384 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_120_120_0
2026.05.06 08:48:34.068720 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_274_274_0 from disk default
2026.05.06 08:48:34.069773 [ 32342 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_119_119_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.069830 [ 32342 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_119_119_0 on disk default
2026.05.06 08:48:34.069896 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_119_119_0
2026.05.06 08:48:34.070037 [ 32342 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_158_158_0 from disk default
2026.05.06 08:48:34.071162 [ 32374 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_120_120_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.071207 [ 32374 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_120_120_0 on disk default
2026.05.06 08:48:34.071277 [ 32374 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_120_120_0
2026.05.06 08:48:34.071496 [ 32374 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_124_124_0 from disk default
2026.05.06 08:48:34.071542 [ 32361 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_125_125_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.071621 [ 32361 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_125_125_0 on disk default
2026.05.06 08:48:34.071710 [ 32361 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_125_125_0
2026.05.06 08:48:34.071875 [ 32377 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_123_123_0 from disk default
2026.05.06 08:48:34.072002 [ 32342 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_158_158_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.072047 [ 32342 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_158_158_0 on disk default
2026.05.06 08:48:34.072109 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_158_158_0
2026.05.06 08:48:34.072277 [ 32373 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_123_123_0 from disk default
2026.05.06 08:48:34.073319 [ 31633 ] {} <Debug> SystemLog: Creating system.crash_log from crash_log
2026.05.06 08:48:34.074366 [ 32377 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_123_123_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.074413 [ 32377 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_123_123_0 on disk default
2026.05.06 08:48:34.074475 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_123_123_0
2026.05.06 08:48:34.074630 [ 32377 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_138_138_0 from disk default
2026.05.06 08:48:34.079886 [ 32374 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_124_124_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.079936 [ 32374 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_124_124_0 on disk default
2026.05.06 08:48:34.079966 [ 32387 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_137_137_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.080004 [ 32374 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_124_124_0
2026.05.06 08:48:34.080028 [ 32387 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_137_137_0 on disk default
2026.05.06 08:48:34.080120 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_137_137_0
2026.05.06 08:48:34.080143 [ 32401 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_274_274_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.080196 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_274_274_0 on disk default
2026.05.06 08:48:34.080268 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_274_274_0
2026.05.06 08:48:34.080748 [ 32344 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_118_118_0 from disk default
2026.05.06 08:48:34.080992 [ 32387 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_141_141_0 from disk default
2026.05.06 08:48:34.081229 [ 32401 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_157_157_0 from disk default
2026.05.06 08:48:34.081714 [ 32377 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_138_138_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.081777 [ 32377 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_138_138_0 on disk default
2026.05.06 08:48:34.084342 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_138_138_0
2026.05.06 08:48:34.084517 [ 32377 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_140_140_0 from disk default
2026.05.06 08:48:34.087560 [ 32401 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_157_157_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.087625 [ 32401 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_157_157_0 on disk default
2026.05.06 08:48:34.087698 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_157_157_0
2026.05.06 08:48:34.087871 [ 32403 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_122_122_0 from disk default
2026.05.06 08:48:34.087974 [ 32338 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_130_130_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.088024 [ 32338 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_130_130_0 on disk default
2026.05.06 08:48:34.088095 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_130_130_0
2026.05.06 08:48:34.088225 [ 32338 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_117_117_0 from disk default
2026.05.06 08:48:34.090965 [ 32403 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_122_122_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.091013 [ 32403 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_122_122_0 on disk default
2026.05.06 08:48:34.091075 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_122_122_0
2026.05.06 08:48:34.091239 [ 32403 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_137_137_0 from disk default
2026.05.06 08:48:34.091675 [ 32373 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_123_123_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.091746 [ 32373 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_123_123_0 on disk default
2026.05.06 08:48:34.091829 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_123_123_0
2026.05.06 08:48:34.091979 [ 32373 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_1_121_24 from disk default
2026.05.06 08:48:34.095804 [ 32344 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_118_118_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.095857 [ 32344 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_118_118_0 on disk default
2026.05.06 08:48:34.096603 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_118_118_0
2026.05.06 08:48:34.096750 [ 32344 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_139_139_0 from disk default
2026.05.06 08:48:34.096872 [ 32373 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_1_121_24 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.096919 [ 32373 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_1_121_24 on disk default
2026.05.06 08:48:34.098848 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_1_121_24
2026.05.06 08:48:34.098975 [ 32373 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_136_136_0 from disk default
2026.05.06 08:48:34.107777 [ 32403 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_137_137_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.107851 [ 32403 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_137_137_0 on disk default
2026.05.06 08:48:34.107926 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Renaming temporary part tmp_merge_202605_1_175_35 to 202605_1_175_35 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:34.107937 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_137_137_0
2026.05.06 08:48:34.108133 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): preparePartForCommit: inserting 202605_1_175_35 (state PreActive) into data_parts_indexes
2026.05.06 08:48:34.108135 [ 32403 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_135_135_0 from disk default
2026.05.06 08:48:34.108197 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f) (MergerMutator): Merged 5 parts: [202605_1_171_34, 202605_175_175_0] -> 202605_1_175_35
2026.05.06 08:48:34.108294 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_1_171_34
2026.05.06 08:48:34.108349 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_172_172_0
2026.05.06 08:48:34.108394 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_173_173_0
2026.05.06 08:48:34.108438 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_174_174_0
2026.05.06 08:48:34.108492 [ 32274 ] {99771344-2c6d-47ba-b88e-32514a4fcc9f::202605_1_175_35} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_175_175_0
2026.05.06 08:48:34.112109 [ 32338 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_117_117_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.112164 [ 32338 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_117_117_0 on disk default
2026.05.06 08:48:34.112234 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_117_117_0
2026.05.06 08:48:34.112367 [ 32338 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_273_273_0 from disk default
2026.05.06 08:48:34.113555 [ 32336 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_130_130_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.113607 [ 32336 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_130_130_0 on disk default
2026.05.06 08:48:34.115043 [ 32336 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_130_130_0
2026.05.06 08:48:34.115301 [ 32367 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_105_105_0 from disk default
2026.05.06 08:48:34.115990 [ 32394 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_142_142_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.116052 [ 32394 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_142_142_0 on disk default
2026.05.06 08:48:34.116119 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_142_142_0
2026.05.06 08:48:34.116251 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_272_272_0 from disk default
2026.05.06 08:48:34.117845 [ 32370 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_128_128_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.117901 [ 32370 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_128_128_0 on disk default
2026.05.06 08:48:34.117967 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_128_128_0
2026.05.06 08:48:34.118139 [ 32370 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_134_134_0 from disk default
2026.05.06 08:48:34.126004 [ 32403 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_135_135_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.126100 [ 32403 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_135_135_0 on disk default
2026.05.06 08:48:34.126191 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_135_135_0
2026.05.06 08:48:34.126367 [ 32373 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_136_136_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.126383 [ 32403 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_119_119_0 from disk default
2026.05.06 08:48:34.126429 [ 32373 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_136_136_0 on disk default
2026.05.06 08:48:34.126523 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_136_136_0
2026.05.06 08:48:34.126667 [ 32373 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_1_126_25 from disk default
2026.05.06 08:48:34.130535 [ 32370 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_134_134_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.130588 [ 32370 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_134_134_0 on disk default
2026.05.06 08:48:34.130700 [ 32338 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_273_273_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.130766 [ 32338 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_273_273_0 on disk default
2026.05.06 08:48:34.132143 [ 32394 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_272_272_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.132228 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_272_272_0 on disk default
2026.05.06 08:48:34.132364 [ 32274 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 6.18 MiB.
2026.05.06 08:48:34.133641 [ 32373 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_1_126_25 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.133676 [ 32398 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_129_129_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.133718 [ 32373 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_1_126_25 on disk default
2026.05.06 08:48:34.133767 [ 32398 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_129_129_0 on disk default
2026.05.06 08:48:34.135005 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_134_134_0
2026.05.06 08:48:34.135146 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_273_273_0
2026.05.06 08:48:34.135195 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_272_272_0
2026.05.06 08:48:34.135228 [ 32398 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_129_129_0
2026.05.06 08:48:34.135233 [ 32370 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_117_121_1 from disk default
2026.05.06 08:48:34.135233 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_1_126_25
2026.05.06 08:48:34.135349 [ 32374 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_104_104_0 from disk default
2026.05.06 08:48:34.135447 [ 32373 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_1_131_26 from disk default
2026.05.06 08:48:34.135523 [ 32390 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_156_156_0 from disk default
2026.05.06 08:48:34.135636 [ 32398 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_116_33 from disk default
2026.05.06 08:48:34.136738 [ 31633 ] {} <Debug> SystemLog: Creating system.text_log from text_log
2026.05.06 08:48:34.137088 [ 32403 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_119_119_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.137155 [ 32403 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_119_119_0 on disk default
2026.05.06 08:48:34.137199 [ 32387 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_141_141_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.137260 [ 32387 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_141_141_0 on disk default
2026.05.06 08:48:34.138145 [ 32390 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_156_156_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.138197 [ 32390 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_156_156_0 on disk default
2026.05.06 08:48:34.138942 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_119_119_0
2026.05.06 08:48:34.138970 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_141_141_0
2026.05.06 08:48:34.138982 [ 32390 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_156_156_0
2026.05.06 08:48:34.139120 [ 32403 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_103_103_0 from disk default
2026.05.06 08:48:34.139196 [ 32401 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_118_118_0 from disk default
2026.05.06 08:48:34.139305 [ 32382 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_117_117_0 from disk default
2026.05.06 08:48:34.139934 [ 32377 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_140_140_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.139982 [ 32377 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_140_140_0 on disk default
2026.05.06 08:48:34.140048 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_140_140_0
2026.05.06 08:48:34.140172 [ 32387 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_102_102_0 from disk default
2026.05.06 08:48:34.141590 [ 32373 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_1_131_26 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.141643 [ 32373 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_1_131_26 on disk default
2026.05.06 08:48:34.143527 [ 32370 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_117_121_1 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.143579 [ 32370 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_117_121_1 on disk default
2026.05.06 08:48:34.144758 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_117_121_1
2026.05.06 08:48:34.144795 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_1_131_26
2026.05.06 08:48:34.144918 [ 32370 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_129_129_0 from disk default
2026.05.06 08:48:34.145165 [ 32373 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_116_116_0 from disk default
2026.05.06 08:48:34.148805 [ 32401 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_118_118_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.148875 [ 32401 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_118_118_0 on disk default
2026.05.06 08:48:34.148959 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_118_118_0
2026.05.06 08:48:34.149184 [ 32369 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_122_122_0 from disk default
2026.05.06 08:48:34.149517 [ 32382 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_117_117_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.149570 [ 32382 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_117_117_0 on disk default
2026.05.06 08:48:34.149636 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_117_117_0
2026.05.06 08:48:34.149793 [ 32382 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_133_133_0 from disk default
2026.05.06 08:48:34.150029 [ 32344 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_139_139_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.150098 [ 32344 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_139_139_0 on disk default
2026.05.06 08:48:34.150191 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_139_139_0
2026.05.06 08:48:34.150352 [ 32344 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_115_115_0 from disk default
2026.05.06 08:48:34.155279 [ 32382 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_133_133_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.155344 [ 32382 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_133_133_0 on disk default
2026.05.06 08:48:34.156067 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_133_133_0
2026.05.06 08:48:34.156276 [ 32382 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_128_128_0 from disk default
2026.05.06 08:48:34.161917 [ 32373 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_116_116_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.161971 [ 32373 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_116_116_0 on disk default
2026.05.06 08:48:34.162464 [ 32344 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_115_115_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.162500 [ 32369 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_122_122_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.162533 [ 32344 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_115_115_0 on disk default
2026.05.06 08:48:34.162553 [ 32369 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_122_122_0 on disk default
2026.05.06 08:48:34.162608 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_115_115_0
2026.05.06 08:48:34.162621 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_122_122_0
2026.05.06 08:48:34.162748 [ 32344 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_1_136_27 from disk default
2026.05.06 08:48:34.162976 [ 32369 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_136_136_0 from disk default
2026.05.06 08:48:34.163762 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_116_116_0
2026.05.06 08:48:34.163972 [ 32337 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_114_114_0 from disk default
2026.05.06 08:48:34.164844 [ 32398 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_116_33 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.164911 [ 32398 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_116_33 on disk default
2026.05.06 08:48:34.165003 [ 32398 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_116_33
2026.05.06 08:48:34.165213 [ 32398 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_155_155_0 from disk default
2026.05.06 08:48:34.167469 [ 32398 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_155_155_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.167538 [ 32398 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_155_155_0 on disk default
2026.05.06 08:48:34.168119 [ 32398 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_155_155_0
2026.05.06 08:48:34.168359 [ 32342 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_154_154_0 from disk default
2026.05.06 08:48:34.168657 [ 32337 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_114_114_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.168727 [ 32337 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_114_114_0 on disk default
2026.05.06 08:48:34.168822 [ 32337 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_114_114_0
2026.05.06 08:48:34.168977 [ 32337 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_153_153_0 from disk default
2026.05.06 08:48:34.169260 [ 32344 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_1_136_27 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.169320 [ 32344 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_1_136_27 on disk default
2026.05.06 08:48:34.169388 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_1_136_27
2026.05.06 08:48:34.169556 [ 32344 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_132_132_0 from disk default
2026.05.06 08:48:34.171700 [ 32337 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_153_153_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.171769 [ 32337 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_153_153_0 on disk default
2026.05.06 08:48:34.171837 [ 32337 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_153_153_0
2026.05.06 08:48:34.172035 [ 32337 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_1_101_20 from disk default
2026.05.06 08:48:34.172079 [ 32342 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_154_154_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.172103 [ 32369 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_136_136_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.172146 [ 32342 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_154_154_0 on disk default
2026.05.06 08:48:34.172164 [ 32369 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_136_136_0 on disk default
2026.05.06 08:48:34.172233 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_154_154_0
2026.05.06 08:48:34.172414 [ 32342 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_1_141_28 from disk default
2026.05.06 08:48:34.174656 [ 32344 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_132_132_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.174722 [ 32344 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_132_132_0 on disk default
2026.05.06 08:48:34.174801 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_132_132_0
2026.05.06 08:48:34.175072 [ 32339 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_152_152_0 from disk default
2026.05.06 08:48:34.175990 [ 32342 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_1_141_28 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.176051 [ 32342 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_1_141_28 on disk default
2026.05.06 08:48:34.176120 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_1_141_28
2026.05.06 08:48:34.176311 [ 32395 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_113_113_0 from disk default
2026.05.06 08:48:34.177102 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_136_136_0
2026.05.06 08:48:34.177239 [ 32369 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_138_138_0 from disk default
2026.05.06 08:48:34.178442 [ 32339 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_152_152_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.178517 [ 32339 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_152_152_0 on disk default
2026.05.06 08:48:34.178597 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_152_152_0
2026.05.06 08:48:34.178815 [ 32339 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_151_151_0 from disk default
2026.05.06 08:48:34.179917 [ 31633 ] {} <Debug> SystemLog: Creating system.metric_log from metric_log
2026.05.06 08:48:34.182811 [ 32339 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_151_151_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.182858 [ 32339 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_151_151_0 on disk default
2026.05.06 08:48:34.183363 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_151_151_0
2026.05.06 08:48:34.183617 [ 32378 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_150_150_0 from disk default
2026.05.06 08:48:34.184838 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 2020 rows, containing 25 columns (25 merged, 0 gathered) in 0.684571196 sec., 2950.7522545543966 rows/sec., 2.40 MiB/sec.
2026.05.06 08:48:34.185455 [ 32395 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_113_113_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.185534 [ 32395 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_113_113_0 on disk default
2026.05.06 08:48:34.185625 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_113_113_0
2026.05.06 08:48:34.185800 [ 32395 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_1_131_26 from disk default
2026.05.06 08:48:34.186691 [ 32378 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_150_150_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.186750 [ 32378 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_150_150_0 on disk default
2026.05.06 08:48:34.186835 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_150_150_0
2026.05.06 08:48:34.186986 [ 32378 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_137_137_0 from disk default
2026.05.06 08:48:34.192447 [ 32395 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_1_131_26 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.192514 [ 32395 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_1_131_26 on disk default
2026.05.06 08:48:34.192737 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_1_131_26
2026.05.06 08:48:34.192908 [ 32395 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_135_135_0 from disk default
2026.05.06 08:48:34.193871 [ 32365 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_127_127_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.193933 [ 32365 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_127_127_0 on disk default
2026.05.06 08:48:34.194027 [ 32365 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_127_127_0
2026.05.06 08:48:34.194203 [ 32365 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_121_34 from disk default
2026.05.06 08:48:34.198698 [ 32395 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_135_135_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.198756 [ 32395 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_135_135_0 on disk default
2026.05.06 08:48:34.198820 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_135_135_0
2026.05.06 08:48:34.198955 [ 32370 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_129_129_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.198997 [ 32395 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_112_25 from disk default
2026.05.06 08:48:34.199027 [ 32370 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_129_129_0 on disk default
2026.05.06 08:48:34.199123 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_129_129_0
2026.05.06 08:48:34.199286 [ 32370 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_149_149_0 from disk default
2026.05.06 08:48:34.203279 [ 32365 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_121_34 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.203355 [ 32365 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_121_34 on disk default
2026.05.06 08:48:34.203451 [ 32365 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_121_34
2026.05.06 08:48:34.203757 [ 32381 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_1_106_21 from disk default
2026.05.06 08:48:34.204543 [ 32370 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_149_149_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.204609 [ 32370 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_149_149_0 on disk default
2026.05.06 08:48:34.204701 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_149_149_0
2026.05.06 08:48:34.204860 [ 32370 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_122_35 from disk default
2026.05.06 08:48:34.208433 [ 32382 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_128_128_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.208504 [ 32382 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_128_128_0 on disk default
2026.05.06 08:48:34.208573 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_128_128_0
2026.05.06 08:48:34.208602 [ 32369 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_138_138_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.208660 [ 32369 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_138_138_0 on disk default
2026.05.06 08:48:34.208759 [ 32382 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_271_271_0 from disk default
2026.05.06 08:48:34.211980 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_138_138_0
2026.05.06 08:48:34.212283 [ 32369 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_134_134_0 from disk default
2026.05.06 08:48:34.213352 [ 32395 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_112_25 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.213418 [ 32395 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_112_25 on disk default
2026.05.06 08:48:34.213512 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_112_25
2026.05.06 08:48:34.213755 [ 32395 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_136_136_0 from disk default
2026.05.06 08:48:34.220405 [ 32378 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_137_137_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.220477 [ 32378 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_137_137_0 on disk default
2026.05.06 08:48:34.220584 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_137_137_0
2026.05.06 08:48:34.220814 [ 32394 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_135_135_0 from disk default
2026.05.06 08:48:34.225133 [ 32370 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_122_35 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.225198 [ 32370 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_122_35 on disk default
2026.05.06 08:48:34.225284 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_122_35
2026.05.06 08:48:34.225447 [ 32370 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_127_127_0 from disk default
2026.05.06 08:48:34.226083 [ 32369 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_134_134_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.226137 [ 32369 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_134_134_0 on disk default
2026.05.06 08:48:34.226229 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_134_134_0
2026.05.06 08:48:34.226393 [ 32369 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_148_148_0 from disk default
2026.05.06 08:48:34.231670 [ 32382 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_271_271_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.231723 [ 32382 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_271_271_0 on disk default
2026.05.06 08:48:34.231790 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_271_271_0
2026.05.06 08:48:34.231961 [ 32382 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_147_147_0 from disk default
2026.05.06 08:48:34.236383 [ 32369 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_148_148_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.236446 [ 32369 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_148_148_0 on disk default
2026.05.06 08:48:34.236535 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_148_148_0
2026.05.06 08:48:34.236711 [ 32369 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_133_133_0 from disk default
2026.05.06 08:48:34.239061 [ 32347 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_124_124_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.239115 [ 32382 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_147_147_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.239202 [ 32382 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_147_147_0 on disk default
2026.05.06 08:48:34.239309 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_147_147_0
2026.05.06 08:48:34.241477 [ 32394 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_135_135_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.241546 [ 32394 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_135_135_0 on disk default
2026.05.06 08:48:34.241615 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_135_135_0
2026.05.06 08:48:34.243026 [ 32343 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_126_126_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.243077 [ 32343 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_126_126_0 on disk default
2026.05.06 08:48:34.243144 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_126_126_0
2026.05.06 08:48:34.243739 [ 32382 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_132_132_0 from disk default
2026.05.06 08:48:34.243988 [ 32339 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_121_121_0 from disk default
2026.05.06 08:48:34.244164 [ 32377 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_126_126_0 from disk default
2026.05.06 08:48:34.239123 [ 32347 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_124_124_0 on disk default
2026.05.06 08:48:34.245353 [ 32347 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_124_124_0
2026.05.06 08:48:34.245597 [ 32392 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_125_125_0 from disk default
2026.05.06 08:48:34.246222 [ 32369 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_133_133_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.246277 [ 32369 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_133_133_0 on disk default
2026.05.06 08:48:34.246341 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_133_133_0
2026.05.06 08:48:34.246477 [ 32369 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_1_146_29 from disk default
2026.05.06 08:48:34.250797 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Trace> MergedBlockOutputStream: filled checksums 202605_301_310_2 (state Temporary)
2026.05.06 08:48:34.252788 [ 32369 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_1_146_29 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.252855 [ 32369 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_1_146_29 on disk default
2026.05.06 08:48:34.252929 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_1_146_29
2026.05.06 08:48:34.253163 [ 32340 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_1_111_22 from disk default
2026.05.06 08:48:34.254169 [ 32368 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_120_120_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.254243 [ 32368 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_120_120_0 on disk default
2026.05.06 08:48:34.254325 [ 32368 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_120_120_0
2026.05.06 08:48:34.254422 [ 32404 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_125_125_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.254497 [ 32404 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_125_125_0 on disk default
2026.05.06 08:48:34.255022 [ 32368 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_123_36 from disk default
2026.05.06 08:48:34.259321 [ 32395 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_136_136_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.259389 [ 32395 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_136_136_0 on disk default
2026.05.06 08:48:34.259659 [ 32382 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_132_132_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.259711 [ 32382 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_132_132_0 on disk default
2026.05.06 08:48:34.259881 [ 32404 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_125_125_0
2026.05.06 08:48:34.260061 [ 32404 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_117_26 from disk default
2026.05.06 08:48:34.261881 [ 32339 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_121_121_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.261942 [ 32339 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_121_121_0 on disk default
2026.05.06 08:48:34.264174 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_136_136_0
2026.05.06 08:48:34.264181 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_132_132_0
2026.05.06 08:48:34.264290 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_121_121_0
2026.05.06 08:48:34.264455 [ 32365 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_1_116_23 from disk default
2026.05.06 08:48:34.264654 [ 32353 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_122_27 from disk default
2026.05.06 08:48:34.264800 [ 32339 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_146_146_0 from disk default
2026.05.06 08:48:34.267590 [ 32339 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_146_146_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.267679 [ 32339 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_146_146_0 on disk default
2026.05.06 08:48:34.267748 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_146_146_0
2026.05.06 08:48:34.267966 [ 32339 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_1_121_24 from disk default
2026.05.06 08:48:34.269794 [ 32353 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_122_27 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.269848 [ 32353 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_122_27 on disk default
2026.05.06 08:48:34.269911 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_122_27
2026.05.06 08:48:34.270042 [ 32353 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loading Outdated part 202605_1_126_25 from disk default
2026.05.06 08:48:34.270912 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Renaming temporary part tmp_merge_202605_301_310_2 to 202605_301_310_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:34.271148 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): preparePartForCommit: inserting 202605_301_310_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:34.271217 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d) (MergerMutator): Merged 5 parts: [202605_301_306_1, 202605_310_310_0] -> 202605_301_310_2
2026.05.06 08:48:34.271328 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_301_306_1
2026.05.06 08:48:34.271385 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_307_307_0
2026.05.06 08:48:34.271432 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_308_308_0
2026.05.06 08:48:34.271478 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_309_309_0
2026.05.06 08:48:34.271538 [ 32279 ] {019282f6-7d19-466e-be33-b02db2b3cf9d::202605_301_310_2} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_310_310_0
2026.05.06 08:48:34.276515 [ 32404 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_117_26 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.276571 [ 32404 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_117_26 on disk default
2026.05.06 08:48:34.276639 [ 32404 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_117_26
2026.05.06 08:48:34.276832 [ 32338 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_131_131_0 from disk default
2026.05.06 08:48:34.277711 [ 32368 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_123_36 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.277777 [ 32368 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_123_36 on disk default
2026.05.06 08:48:34.277854 [ 32368 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_123_36
2026.05.06 08:48:34.277984 [ 32368 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_1_136_27 from disk default
2026.05.06 08:48:34.285763 [ 32338 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_131_131_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.285822 [ 32338 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_131_131_0 on disk default
2026.05.06 08:48:34.285887 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_131_131_0
2026.05.06 08:48:34.286025 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_120_120_0 from disk default
2026.05.06 08:48:34.291121 [ 32279 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 8.76 MiB.
2026.05.06 08:48:34.295763 [ 32368 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_1_136_27 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.295846 [ 32368 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_1_136_27 on disk default
2026.05.06 08:48:34.295947 [ 32368 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_1_136_27
2026.05.06 08:48:34.296127 [ 32368 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_134_134_0 from disk default
2026.05.06 08:48:34.297151 [ 32392 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_125_125_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.297228 [ 32392 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_125_125_0 on disk default
2026.05.06 08:48:34.297323 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_125_125_0
2026.05.06 08:48:34.297546 [ 32344 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_270_270_0 from disk default
2026.05.06 08:48:34.298984 [ 32370 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_127_127_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.299065 [ 32370 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_127_127_0 on disk default
2026.05.06 08:48:34.299159 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_127_127_0
2026.05.06 08:48:34.299309 [ 32370 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_124_37 from disk default
2026.05.06 08:48:34.299991 [ 32338 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_120_120_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.300044 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_120_120_0 on disk default
2026.05.06 08:48:34.300108 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_120_120_0
2026.05.06 08:48:34.300291 [ 32390 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_124_28 from disk default
2026.05.06 08:48:34.311060 [ 32377 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_126_126_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.311119 [ 32377 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_126_126_0 on disk default
2026.05.06 08:48:34.311395 [ 32390 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_124_28 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.311446 [ 32390 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_124_28 on disk default
2026.05.06 08:48:34.312039 [ 32344 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_270_270_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.312152 [ 32344 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_270_270_0 on disk default
2026.05.06 08:48:34.313933 [ 32385 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_123_123_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.314014 [ 32385 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_123_123_0 on disk default
2026.05.06 08:48:34.314972 [ 32370 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_124_37 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.315037 [ 32370 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_124_37 on disk default
2026.05.06 08:48:34.318032 [ 32390 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_124_28
2026.05.06 08:48:34.318060 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_270_270_0
2026.05.06 08:48:34.318032 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_126_126_0
2026.05.06 08:48:34.318098 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_124_37
2026.05.06 08:48:34.318067 [ 32385 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_123_123_0
2026.05.06 08:48:34.318244 [ 32394 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_133_133_0 from disk default
2026.05.06 08:48:34.318303 [ 32377 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_145_145_0 from disk default
2026.05.06 08:48:34.318348 [ 32344 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_144_144_0 from disk default
2026.05.06 08:48:34.318409 [ 32343 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_143_143_0 from disk default
2026.05.06 08:48:34.318882 [ 32385 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_142_142_0 from disk default
2026.05.06 08:48:34.321161 [ 32343 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_143_143_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.321230 [ 32343 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_143_143_0 on disk default
2026.05.06 08:48:34.321751 [ 32344 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_144_144_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.321805 [ 32344 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_144_144_0 on disk default
2026.05.06 08:48:34.322175 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_144_144_0
2026.05.06 08:48:34.322210 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_143_143_0
2026.05.06 08:48:34.322368 [ 32341 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_141_141_0 from disk default
2026.05.06 08:48:34.322473 [ 32343 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_124_124_0 from disk default
2026.05.06 08:48:34.324325 [ 32377 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_145_145_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.324387 [ 32377 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_145_145_0 on disk default
2026.05.06 08:48:34.324474 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_145_145_0
2026.05.06 08:48:34.324640 [ 32377 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_140_140_0 from disk default
2026.05.06 08:48:34.324831 [ 32385 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_142_142_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.324885 [ 32385 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_142_142_0 on disk default
2026.05.06 08:48:34.324967 [ 32385 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_142_142_0
2026.05.06 08:48:34.325153 [ 32385 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_269_269_0 from disk default
2026.05.06 08:48:34.325582 [ 32341 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_141_141_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.325632 [ 32341 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_141_141_0 on disk default
2026.05.06 08:48:34.325699 [ 32341 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_141_141_0
2026.05.06 08:48:34.325887 [ 32373 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_119_119_0 from disk default
2026.05.06 08:48:34.326644 [ 32377 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_140_140_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.326691 [ 32377 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_140_140_0 on disk default
2026.05.06 08:48:34.327742 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_140_140_0
2026.05.06 08:48:34.327935 [ 32361 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_1_141_28 from disk default
2026.05.06 08:48:34.331618 [ 32361 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_1_141_28 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.331675 [ 32361 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_1_141_28 on disk default
2026.05.06 08:48:34.331738 [ 32361 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_1_141_28
2026.05.06 08:48:34.331863 [ 32377 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_1_146_29 from disk default
2026.05.06 08:48:34.331996 [ 32394 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_133_133_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.332054 [ 32394 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_133_133_0 on disk default
2026.05.06 08:48:34.332126 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_133_133_0
2026.05.06 08:48:34.332397 [ 32394 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_139_139_0 from disk default
2026.05.06 08:48:34.338186 [ 32394 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_139_139_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.338240 [ 32394 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_139_139_0 on disk default
2026.05.06 08:48:34.338308 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_139_139_0
2026.05.06 08:48:34.338351 [ 32373 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_119_119_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.338399 [ 32373 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_119_119_0 on disk default
2026.05.06 08:48:34.338442 [ 32394 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_1_151_30 from disk default
2026.05.06 08:48:34.338460 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_119_119_0
2026.05.06 08:48:34.338592 [ 32373 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_132_132_0 from disk default
2026.05.06 08:48:34.339925 [ 32385 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_269_269_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.339991 [ 32385 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_269_269_0 on disk default
2026.05.06 08:48:34.340093 [ 32385 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_269_269_0
2026.05.06 08:48:34.340330 [ 32380 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_125_29 from disk default
2026.05.06 08:48:34.340561 [ 32368 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_134_134_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.340619 [ 32368 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_134_134_0 on disk default
2026.05.06 08:48:34.340687 [ 32368 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_134_134_0
2026.05.06 08:48:34.340885 [ 32392 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_1_156_31 from disk default
2026.05.06 08:48:34.341166 [ 32394 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_1_151_30 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.341235 [ 32394 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_1_151_30 on disk default
2026.05.06 08:48:34.341314 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_1_151_30
2026.05.06 08:48:34.341435 [ 32394 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_125_38 from disk default
2026.05.06 08:48:34.343834 [ 32377 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_1_146_29 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.343894 [ 32377 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_1_146_29 on disk default
2026.05.06 08:48:34.343966 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_1_146_29
2026.05.06 08:48:34.344127 [ 32377 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_118_118_0 from disk default
2026.05.06 08:48:34.345797 [ 32392 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_1_156_31 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.345869 [ 32392 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_1_156_31 on disk default
2026.05.06 08:48:34.345956 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_1_156_31
2026.05.06 08:48:34.346141 [ 32392 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_126_39 from disk default
2026.05.06 08:48:34.346412 [ 32380 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_125_29 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.346497 [ 32380 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_125_29 on disk default
2026.05.06 08:48:34.347902 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_125_29
2026.05.06 08:48:34.348083 [ 32380 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_117_117_0 from disk default
2026.05.06 08:48:34.349175 [ 32402 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_122_122_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.349259 [ 32402 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_122_122_0 on disk default
2026.05.06 08:48:34.349351 [ 32402 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_122_122_0
2026.05.06 08:48:34.349384 [ 32394 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_125_38 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.349456 [ 32394 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_125_38 on disk default
2026.05.06 08:48:34.349531 [ 32402 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_1_151_30 from disk default
2026.05.06 08:48:34.349554 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_125_38
2026.05.06 08:48:34.349699 [ 32394 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loading Outdated part 202605_1_161_32 from disk default
2026.05.06 08:48:34.355426 [ 32392 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_126_39 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.355496 [ 32392 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_126_39 on disk default
2026.05.06 08:48:34.355570 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_126_39
2026.05.06 08:48:34.355730 [ 32392 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_123_123_0 from disk default
2026.05.06 08:48:34.362821 [ 32402 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_1_151_30 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.362879 [ 32402 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_1_151_30 on disk default
2026.05.06 08:48:34.362951 [ 32402 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_1_151_30
2026.05.06 08:48:34.363098 [ 32402 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_138_138_0 from disk default
2026.05.06 08:48:34.364511 [ 32394 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): loadDataPart: inserting 202605_1_161_32 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.364577 [ 32394 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Finished loading Outdated part 202605_1_161_32 on disk default
2026.05.06 08:48:34.365463 [ 32343 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_124_124_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.365530 [ 32343 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_124_124_0 on disk default
2026.05.06 08:48:34.366953 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_1_161_32
2026.05.06 08:48:34.366981 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_124_124_0
2026.05.06 08:48:34.367153 [ 32343 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_1_156_31 from disk default
2026.05.06 08:48:34.368136 [ 32380 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_117_117_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.368205 [ 32380 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_117_117_0 on disk default
2026.05.06 08:48:34.368298 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_117_117_0
2026.05.06 08:48:34.368478 [ 32380 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_1_161_32 from disk default
2026.05.06 08:48:34.369898 [ 32360 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_121_121_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.369991 [ 32360 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_121_121_0 on disk default
2026.05.06 08:48:34.370096 [ 32360 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_121_121_0
2026.05.06 08:48:34.370274 [ 32360 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loading Outdated part 202605_1_166_33 from disk default
2026.05.06 08:48:34.370523 [ 32377 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_118_118_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.370587 [ 32377 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_118_118_0 on disk default
2026.05.06 08:48:34.370673 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_118_118_0
2026.05.06 08:48:34.370885 [ 31696 ] {} <Debug> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Loaded 54 outdated data parts asynchronously
2026.05.06 08:48:34.370951 [ 31696 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 950 ms.
2026.05.06 08:48:34.371005 [ 32394 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_123_128_1 from disk default
2026.05.06 08:48:34.371253 [ 32377 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_137_137_0 from disk default
2026.05.06 08:48:34.373457 [ 32380 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_1_161_32 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.373468 [ 32373 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_132_132_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.373537 [ 32373 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_132_132_0 on disk default
2026.05.06 08:48:34.373542 [ 32380 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_1_161_32 on disk default
2026.05.06 08:48:34.373613 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_132_132_0
2026.05.06 08:48:34.373630 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_1_161_32
2026.05.06 08:48:34.373757 [ 32373 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_131_131_0 from disk default
2026.05.06 08:48:34.373820 [ 32380 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_123_133_2 from disk default
2026.05.06 08:48:34.374605 [ 32375 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_119_119_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.374678 [ 32375 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_119_119_0 on disk default
2026.05.06 08:48:34.374750 [ 32375 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_119_119_0
2026.05.06 08:48:34.374888 [ 32375 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_1_136_27 from disk default
2026.05.06 08:48:34.374925 [ 32402 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_138_138_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.375002 [ 32402 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_138_138_0 on disk default
2026.05.06 08:48:34.375080 [ 32402 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_138_138_0
2026.05.06 08:48:34.375234 [ 32402 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_126_30 from disk default
2026.05.06 08:48:34.380366 [ 32377 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_137_137_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.380423 [ 32377 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_137_137_0 on disk default
2026.05.06 08:48:34.380499 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_137_137_0
2026.05.06 08:48:34.380645 [ 32377 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_130_130_0 from disk default
2026.05.06 08:48:34.382761 [ 32360 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_1_166_33 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.382814 [ 32360 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_1_166_33 on disk default
2026.05.06 08:48:34.382878 [ 32360 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_1_166_33
2026.05.06 08:48:34.385435 [ 32368 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_130_130_0 from disk default
2026.05.06 08:48:34.386948 [ 32343 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): loadDataPart: inserting 202605_1_156_31 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.387010 [ 32343 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Finished loading Outdated part 202605_1_156_31 on disk default
2026.05.06 08:48:34.387082 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f), part: 202605_1_156_31
2026.05.06 08:48:34.388250 [ 31694 ] {} <Debug> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Loaded 48 outdated data parts asynchronously
2026.05.06 08:48:34.388265 [ 32343 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_123_138_3 from disk default
2026.05.06 08:48:34.388331 [ 31694 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 968 ms.
2026.05.06 08:48:34.389124 [ 32351 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_118_118_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.389197 [ 32351 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_118_118_0 on disk default
2026.05.06 08:48:34.389280 [ 32351 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_118_118_0
2026.05.06 08:48:34.389423 [ 32351 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_268_268_0 from disk default
2026.05.06 08:48:34.389605 [ 32375 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_1_136_27 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.389656 [ 32375 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_1_136_27 on disk default
2026.05.06 08:48:34.397906 [ 32375 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_1_136_27
2026.05.06 08:48:34.398125 [ 32375 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_267_267_0 from disk default
2026.05.06 08:48:34.399931 [ 32402 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_126_30 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.400008 [ 32402 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_126_30 on disk default
2026.05.06 08:48:34.400096 [ 32402 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_126_30
2026.05.06 08:48:34.400377 [ 32347 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_266_266_0 from disk default
2026.05.06 08:48:34.405083 [ 32372 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_115_115_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.405170 [ 32372 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_115_115_0 on disk default
2026.05.06 08:48:34.405260 [ 32372 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_115_115_0
2026.05.06 08:48:34.405513 [ 32370 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_265_265_0 from disk default
2026.05.06 08:48:34.405781 [ 32375 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_267_267_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.405781 [ 32377 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_130_130_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.405846 [ 32375 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_267_267_0 on disk default
2026.05.06 08:48:34.405862 [ 32351 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_268_268_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.405915 [ 32375 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_267_267_0
2026.05.06 08:48:34.405912 [ 32377 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_130_130_0 on disk default
2026.05.06 08:48:34.405920 [ 32351 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_268_268_0 on disk default
2026.05.06 08:48:34.406020 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_130_130_0
2026.05.06 08:48:34.406067 [ 32375 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_116_116_0 from disk default
2026.05.06 08:48:34.406166 [ 32377 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_115_115_0 from disk default
2026.05.06 08:48:34.406195 [ 32368 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_130_130_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.406280 [ 32368 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_130_130_0 on disk default
2026.05.06 08:48:34.406860 [ 32351 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_268_268_0
2026.05.06 08:48:34.407001 [ 32351 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_127_31 from disk default
2026.05.06 08:48:34.408502 [ 32396 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_117_117_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.408579 [ 32396 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_117_117_0 on disk default
2026.05.06 08:48:34.409669 [ 32347 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_266_266_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.409777 [ 32347 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_266_266_0 on disk default
2026.05.06 08:48:34.410071 [ 32394 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_123_128_1 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.410149 [ 32394 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_123_128_1 on disk default
2026.05.06 08:48:34.410756 [ 32368 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_130_130_0
2026.05.06 08:48:34.410966 [ 32336 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_129_129_0 from disk default
2026.05.06 08:48:34.411810 [ 32370 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_265_265_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.411828 [ 32396 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_117_117_0
2026.05.06 08:48:34.411876 [ 32370 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_265_265_0 on disk default
2026.05.06 08:48:34.411899 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_123_128_1
2026.05.06 08:48:34.411905 [ 32347 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_266_266_0
2026.05.06 08:48:34.411955 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_265_265_0
2026.05.06 08:48:34.412130 [ 32382 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_128_128_0 from disk default
2026.05.06 08:48:34.412502 [ 32341 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_264_264_0 from disk default
2026.05.06 08:48:34.412613 [ 32361 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_123_143_4 from disk default
2026.05.06 08:48:34.412653 [ 32370 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_263_263_0 from disk default
2026.05.06 08:48:34.414092 [ 32373 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_131_131_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.414178 [ 32373 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_131_131_0 on disk default
2026.05.06 08:48:34.414284 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_131_131_0
2026.05.06 08:48:34.414557 [ 32404 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_127_40 from disk default
2026.05.06 08:48:34.414991 [ 32336 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_129_129_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.415058 [ 32336 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_129_129_0 on disk default
2026.05.06 08:48:34.415136 [ 32336 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_129_129_0
2026.05.06 08:48:34.415307 [ 32378 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_128_32 from disk default
2026.05.06 08:48:34.417637 [ 32375 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_116_116_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.417716 [ 32375 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_116_116_0 on disk default
2026.05.06 08:48:34.417810 [ 32375 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_116_116_0
2026.05.06 08:48:34.418006 [ 32349 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_129_33 from disk default
2026.05.06 08:48:34.418598 [ 32382 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_128_128_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.418657 [ 32382 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_128_128_0 on disk default
2026.05.06 08:48:34.418915 [ 32351 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_127_31 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.418992 [ 32351 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_127_31 on disk default
2026.05.06 08:48:34.419633 [ 32341 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_264_264_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.419702 [ 32341 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_264_264_0 on disk default
2026.05.06 08:48:34.419749 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_128_128_0
2026.05.06 08:48:34.419777 [ 32341 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_264_264_0
2026.05.06 08:48:34.419905 [ 32382 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_1_141_28 from disk default
2026.05.06 08:48:34.421239 [ 32404 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_127_40 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.421300 [ 32404 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_127_40 on disk default
2026.05.06 08:48:34.421364 [ 32404 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_127_40
2026.05.06 08:48:34.421627 [ 32342 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_1_146_29 from disk default
2026.05.06 08:48:34.421635 [ 32378 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_128_32 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.421696 [ 32360 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_127_127_0 from disk default
2026.05.06 08:48:34.421731 [ 32378 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_128_32 on disk default
2026.05.06 08:48:34.421805 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_128_32
2026.05.06 08:48:34.421920 [ 32347 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_123_148_5 from disk default
2026.05.06 08:48:34.422318 [ 32392 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_123_123_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.422374 [ 32392 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_123_123_0 on disk default
2026.05.06 08:48:34.422461 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_123_123_0
2026.05.06 08:48:34.422627 [ 32392 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_114_114_0 from disk default
2026.05.06 08:48:34.422741 [ 32349 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_129_33 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.422798 [ 32349 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_129_33 on disk default
2026.05.06 08:48:34.424700 [ 32382 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_1_141_28 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.424763 [ 32382 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_1_141_28 on disk default
2026.05.06 08:48:34.425640 [ 32342 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_1_146_29 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.425705 [ 32342 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_1_146_29 on disk default
2026.05.06 08:48:34.426476 [ 32380 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_123_133_2 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.426565 [ 32380 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_123_133_2 on disk default
2026.05.06 08:48:34.422623 [ 32351 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_127_31
2026.05.06 08:48:34.426781 [ 32351 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_1_151_30 from disk default
2026.05.06 08:48:34.427594 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_129_33
2026.05.06 08:48:34.427771 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_1_146_29
2026.05.06 08:48:34.427796 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_1_141_28
2026.05.06 08:48:34.427820 [ 32349 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_1_156_31 from disk default
2026.05.06 08:48:34.427904 [ 32342 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_129_129_0 from disk default
2026.05.06 08:48:34.427983 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_123_133_2
2026.05.06 08:48:34.428052 [ 32382 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_1_161_32 from disk default
2026.05.06 08:48:34.428133 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_126_58 from disk default
2026.05.06 08:48:34.431306 [ 32360 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_127_127_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.431372 [ 32360 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_127_127_0 on disk default
2026.05.06 08:48:34.431538 [ 32360 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_127_127_0
2026.05.06 08:48:34.431840 [ 32344 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_128_128_0 from disk default
2026.05.06 08:48:34.431916 [ 32343 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_123_138_3 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.431987 [ 32343 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_123_138_3 on disk default
2026.05.06 08:48:34.432086 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_123_138_3
2026.05.06 08:48:34.432100 [ 32370 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_263_263_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.432184 [ 32370 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_263_263_0 on disk default
2026.05.06 08:48:34.432248 [ 32343 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_127_59 from disk default
2026.05.06 08:48:34.432284 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_263_263_0
2026.05.06 08:48:34.433253 [ 32370 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_128_41 from disk default
2026.05.06 08:48:34.435859 [ 32349 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_1_156_31 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.435932 [ 32349 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_1_156_31 on disk default
2026.05.06 08:48:34.436020 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_1_156_31
2026.05.06 08:48:34.436061 [ 32382 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_1_161_32 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.436117 [ 32382 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_1_161_32 on disk default
2026.05.06 08:48:34.436155 [ 32351 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_1_151_30 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.436197 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_1_161_32
2026.05.06 08:48:34.436211 [ 32351 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_1_151_30 on disk default
2026.05.06 08:48:34.436230 [ 32398 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_129_42 from disk default
2026.05.06 08:48:34.436293 [ 32351 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_1_151_30
2026.05.06 08:48:34.437599 [ 32379 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_114_114_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.437681 [ 32379 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_114_114_0 on disk default
2026.05.06 08:48:34.437850 [ 32379 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_114_114_0
2026.05.06 08:48:34.439030 [ 32380 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_126_58 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.439114 [ 32380 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_126_58 on disk default
2026.05.06 08:48:34.439211 [ 32380 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_126_58
2026.05.06 08:48:34.439505 [ 32343 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_127_59 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.439575 [ 32343 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_127_59 on disk default
2026.05.06 08:48:34.439678 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_127_59
2026.05.06 08:48:34.440837 [ 32398 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_129_42 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.440907 [ 32398 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_129_42 on disk default
2026.05.06 08:48:34.441000 [ 32398 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_129_42
2026.05.06 08:48:34.441166 [ 32395 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_128_60 from disk default
2026.05.06 08:48:34.441362 [ 32375 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_129_61 from disk default
2026.05.06 08:48:34.441402 [ 32385 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_130_62 from disk default
2026.05.06 08:48:34.441441 [ 32394 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_1_166_33 from disk default
2026.05.06 08:48:34.441468 [ 32373 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_113_113_0 from disk default
2026.05.06 08:48:34.441509 [ 32349 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_131_63 from disk default
2026.05.06 08:48:34.442645 [ 32370 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_128_41 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.442709 [ 32370 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_128_41 on disk default
2026.05.06 08:48:34.442816 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_128_41
2026.05.06 08:48:34.443020 [ 32341 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_130_34 from disk default
2026.05.06 08:48:34.443549 [ 32344 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_128_128_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.443599 [ 32344 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_128_128_0 on disk default
2026.05.06 08:48:34.443691 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_128_128_0
2026.05.06 08:48:34.443871 [ 32336 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_131_35 from disk default
2026.05.06 08:48:34.446875 [ 32394 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_1_166_33 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.446964 [ 32394 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_1_166_33 on disk default
2026.05.06 08:48:34.447070 [ 32373 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_113_113_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.447077 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_1_166_33
2026.05.06 08:48:34.447134 [ 32373 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_113_113_0 on disk default
2026.05.06 08:48:34.447207 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_113_113_0
2026.05.06 08:48:34.447269 [ 32394 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_132_36 from disk default
2026.05.06 08:48:34.447364 [ 32378 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_130_43 from disk default
2026.05.06 08:48:34.448091 [ 32336 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_131_35 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.448157 [ 32336 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_131_35 on disk default
2026.05.06 08:48:34.448234 [ 32336 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_131_35
2026.05.06 08:48:34.448343 [ 32341 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_130_34 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.448400 [ 32341 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_130_34 on disk default
2026.05.06 08:48:34.448543 [ 32341 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_130_34
2026.05.06 08:48:34.448692 [ 32349 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_131_63 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.448789 [ 32385 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_130_62 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.448851 [ 32385 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_130_62 on disk default
2026.05.06 08:48:34.449000 [ 32385 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_130_62
2026.05.06 08:48:34.449073 [ 32395 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_128_60 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.449140 [ 32395 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_128_60 on disk default
2026.05.06 08:48:34.449224 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_128_60
2026.05.06 08:48:34.449281 [ 32375 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_129_61 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.449337 [ 32375 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_129_61 on disk default
2026.05.06 08:48:34.449419 [ 32375 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_129_61
2026.05.06 08:48:34.449495 [ 32349 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_131_63 on disk default
2026.05.06 08:48:34.449566 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_131_63
2026.05.06 08:48:34.449843 [ 32336 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_127_127_0 from disk default
2026.05.06 08:48:34.450301 [ 32360 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_132_64 from disk default
2026.05.06 08:48:34.451339 [ 32361 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_123_143_4 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.451410 [ 32361 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_123_143_4 on disk default
2026.05.06 08:48:34.451436 [ 32402 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_262_262_0 from disk default
2026.05.06 08:48:34.451451 [ 32369 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_131_44 from disk default
2026.05.06 08:48:34.451501 [ 32361 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_123_143_4
2026.05.06 08:48:34.451527 [ 32401 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_1_171_34 from disk default
2026.05.06 08:48:34.451717 [ 32396 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_122_122_0 from disk default
2026.05.06 08:48:34.452447 [ 32372 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_138_65 from disk default
2026.05.06 08:48:34.452475 [ 32378 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_130_43 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.452573 [ 32378 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_130_43 on disk default
2026.05.06 08:48:34.454921 [ 32394 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_132_36 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.454999 [ 32394 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_132_36 on disk default
2026.05.06 08:48:34.455087 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_132_36
2026.05.06 08:48:34.455236 [ 32394 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_144_66 from disk default
2026.05.06 08:48:34.456638 [ 32369 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_131_44 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.456719 [ 32369 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_131_44 on disk default
2026.05.06 08:48:34.456821 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_131_44
2026.05.06 08:48:34.457642 [ 32342 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_129_129_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.457714 [ 32342 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_129_129_0 on disk default
2026.05.06 08:48:34.457798 [ 32401 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_1_171_34 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.457808 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_129_129_0
2026.05.06 08:48:34.457903 [ 32401 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_1_171_34 on disk default
2026.05.06 08:48:34.458027 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_1_171_34
2026.05.06 08:48:34.458655 [ 32377 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_115_115_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.458729 [ 32377 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_115_115_0 on disk default
2026.05.06 08:48:34.458825 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_115_115_0
2026.05.06 08:48:34.459004 [ 32369 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_150_67 from disk default
2026.05.06 08:48:34.459784 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_261_261_0 from disk default
2026.05.06 08:48:34.461106 [ 32377 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_260_260_0 from disk default
2026.05.06 08:48:34.461196 [ 32342 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_259_259_0 from disk default
2026.05.06 08:48:34.462718 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_130_43
2026.05.06 08:48:34.462896 [ 32378 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_258_258_0 from disk default
2026.05.06 08:48:34.463369 [ 32392 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_114_114_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.463442 [ 32392 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_114_114_0 on disk default
2026.05.06 08:48:34.463555 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_114_114_0
2026.05.06 08:48:34.463562 [ 32360 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_132_64 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.463642 [ 32360 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_132_64 on disk default
2026.05.06 08:48:34.463727 [ 32360 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_132_64
2026.05.06 08:48:34.463743 [ 32394 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_144_66 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.463810 [ 32394 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_144_66 on disk default
2026.05.06 08:48:34.463899 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_144_66
2026.05.06 08:48:34.464311 [ 32392 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_121_121_0 from disk default
2026.05.06 08:48:34.466182 [ 32369 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_150_67 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.466255 [ 32369 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_150_67 on disk default
2026.05.06 08:48:34.466329 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_150_67
2026.05.06 08:48:34.467799 [ 32344 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_132_45 from disk default
2026.05.06 08:48:34.467875 [ 32402 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_262_262_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.467930 [ 32402 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_262_262_0 on disk default
2026.05.06 08:48:34.467941 [ 32369 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_126_126_0 from disk default
2026.05.06 08:48:34.467969 [ 32394 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_112_112_0 from disk default
2026.05.06 08:48:34.470164 [ 32402 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_262_262_0
2026.05.06 08:48:34.471681 [ 32402 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_120_120_0 from disk default
2026.05.06 08:48:34.473053 [ 32372 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_138_65 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.473108 [ 32372 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_138_65 on disk default
2026.05.06 08:48:34.473118 [ 32378 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_258_258_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.473169 [ 32378 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_258_258_0 on disk default
2026.05.06 08:48:34.473177 [ 32372 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_138_65
2026.05.06 08:48:34.473169 [ 32401 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_261_261_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.473244 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_261_261_0 on disk default
2026.05.06 08:48:34.473248 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_258_258_0
2026.05.06 08:48:34.473329 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_261_261_0
2026.05.06 08:48:34.473340 [ 32372 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loading Outdated part 202605_1_176_35 from disk default
2026.05.06 08:48:34.473440 [ 32378 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_119_119_0 from disk default
2026.05.06 08:48:34.473589 [ 32385 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_125_36 from disk default
2026.05.06 08:48:34.474317 [ 32376 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_116_116_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.474404 [ 32376 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_116_116_0 on disk default
2026.05.06 08:48:34.477390 [ 32376 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_116_116_0
2026.05.06 08:48:34.477582 [ 32311 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Found 48 old parts to remove. Parts: [202605_1_131_26, 202605_1_136_27, 202605_1_141_28, 202605_1_146_29, 202605_1_151_30, 202605_1_156_31, 202605_1_161_32, 202605_1_166_33, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0, 202605_167_167_0, 202605_168_168_0, 202605_169_169_0, 202605_170_170_0, 202605_171_171_0]
2026.05.06 08:48:34.477968 [ 32376 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_133_37 from disk default
2026.05.06 08:48:34.478765 [ 32344 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_132_45 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.478827 [ 32344 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_132_45 on disk default
2026.05.06 08:48:34.478893 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_132_45
2026.05.06 08:48:34.479010 [ 32344 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_118_118_0 from disk default
2026.05.06 08:48:34.482736 [ 32342 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_259_259_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.482803 [ 32342 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_259_259_0 on disk default
2026.05.06 08:48:34.482902 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_259_259_0
2026.05.06 08:48:34.483043 [ 32342 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_111_111_0 from disk default
2026.05.06 08:48:34.480294 [ 32311 ] {} <Debug> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Removing 48 parts from filesystem (serially): Parts: [202605_1_131_26, 202605_1_136_27, 202605_1_141_28, 202605_1_146_29, 202605_1_151_30, 202605_1_156_31, 202605_1_161_32, 202605_1_166_33, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0, 202605_167_167_0, 202605_168_168_0, 202605_169_169_0, 202605_170_170_0, 202605_171_171_0]
2026.05.06 08:48:34.486511 [ 32377 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_260_260_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.486570 [ 32377 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_260_260_0 on disk default
2026.05.06 08:48:34.486645 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_260_260_0
2026.05.06 08:48:34.486795 [ 32377 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_138_38 from disk default
2026.05.06 08:48:34.487028 [ 32312 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Found 48 old parts to remove. Parts: [202605_258_258_0, 202605_259_259_0, 202605_261_261_0, 202605_262_262_0, 202605_263_263_0, 202605_264_264_0, 202605_265_265_0, 202605_266_266_0, 202605_267_267_0, 202605_268_268_0, 202605_269_269_0, 202605_270_270_0, 202605_271_271_0, 202605_272_272_0, 202605_273_273_0, 202605_274_274_0, 202605_275_275_0, 202605_276_276_0, 202605_277_277_0, 202605_278_278_0, 202605_279_279_0, 202605_280_280_0, 202605_281_281_0, 202605_282_282_0, 202605_283_283_0, 202605_284_284_0, 202605_285_285_0, 202605_286_286_0, 202605_287_287_0, 202605_288_288_0, 202605_289_289_0, 202605_290_290_0, 202605_291_291_0, 202605_292_292_0, 202605_293_293_0, 202605_294_294_0, 202605_295_295_0, 202605_296_296_0, 202605_297_297_0, 202605_298_298_0, 202605_299_299_0, 202605_300_300_0, 202605_301_301_0, 202605_302_302_0, 202605_303_303_0, 202605_304_304_0, 202605_305_305_0, 202605_306_306_0]
2026.05.06 08:48:34.487209 [ 32312 ] {} <Debug> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Removing 48 parts from filesystem (serially): Parts: [202605_258_258_0, 202605_259_259_0, 202605_261_261_0, 202605_262_262_0, 202605_263_263_0, 202605_264_264_0, 202605_265_265_0, 202605_266_266_0, 202605_267_267_0, 202605_268_268_0, 202605_269_269_0, 202605_270_270_0, 202605_271_271_0, 202605_272_272_0, 202605_273_273_0, 202605_274_274_0, 202605_275_275_0, 202605_276_276_0, 202605_277_277_0, 202605_278_278_0, 202605_279_279_0, 202605_280_280_0, 202605_281_281_0, 202605_282_282_0, 202605_283_283_0, 202605_284_284_0, 202605_285_285_0, 202605_286_286_0, 202605_287_287_0, 202605_288_288_0, 202605_289_289_0, 202605_290_290_0, 202605_291_291_0, 202605_292_292_0, 202605_293_293_0, 202605_294_294_0, 202605_295_295_0, 202605_296_296_0, 202605_297_297_0, 202605_298_298_0, 202605_299_299_0, 202605_300_300_0, 202605_301_301_0, 202605_302_302_0, 202605_303_303_0, 202605_304_304_0, 202605_305_305_0, 202605_306_306_0]
2026.05.06 08:48:34.488823 [ 32372 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): loadDataPart: inserting 202605_1_176_35 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.488884 [ 32372 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Finished loading Outdated part 202605_1_176_35 on disk default
2026.05.06 08:48:34.488980 [ 32372 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_1_176_35
2026.05.06 08:48:34.489327 [ 31706 ] {} <Debug> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Loaded 54 outdated data parts asynchronously
2026.05.06 08:48:34.489390 [ 31706 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 1067 ms.
2026.05.06 08:48:34.489611 [ 32341 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_138_46 from disk default
2026.05.06 08:48:34.491679 [ 32377 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_138_38 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.491759 [ 32377 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_138_38 on disk default
2026.05.06 08:48:34.491868 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_138_38
2026.05.06 08:48:34.492064 [ 32377 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_110_110_0 from disk default
2026.05.06 08:48:34.492171 [ 32400 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_113_113_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.492262 [ 32400 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_113_113_0 on disk default
2026.05.06 08:48:34.492403 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_113_113_0
2026.05.06 08:48:34.493301 [ 32341 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_138_46 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.493361 [ 32341 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_138_46 on disk default
2026.05.06 08:48:34.493476 [ 32341 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_138_46
2026.05.06 08:48:34.493658 [ 32341 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_126_37 from disk default
2026.05.06 08:48:34.493739 [ 32400 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_131_38 from disk default
2026.05.06 08:48:34.495529 [ 32385 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_125_36 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.495618 [ 32385 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_125_36 on disk default
2026.05.06 08:48:34.495713 [ 32385 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_125_36
2026.05.06 08:48:34.495920 [ 32385 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_136_39 from disk default
2026.05.06 08:48:34.497315 [ 32394 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_112_112_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.497395 [ 32394 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_112_112_0 on disk default
2026.05.06 08:48:34.497501 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_112_112_0
2026.05.06 08:48:34.497658 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_257_257_0 from disk default
2026.05.06 08:48:34.501165 [ 32336 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_127_127_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.501239 [ 32336 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_127_127_0 on disk default
2026.05.06 08:48:34.501331 [ 32336 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_127_127_0
2026.05.06 08:48:34.501553 [ 32336 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_256_256_0 from disk default
2026.05.06 08:48:34.504658 [ 32376 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_133_37 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.504735 [ 32376 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_133_37 on disk default
2026.05.06 08:48:34.506148 [ 32376 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_133_37
2026.05.06 08:48:34.506378 [ 32343 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_255_255_0 from disk default
2026.05.06 08:48:34.507582 [ 32336 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_256_256_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.507657 [ 32336 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_256_256_0 on disk default
2026.05.06 08:48:34.507746 [ 32336 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_256_256_0
2026.05.06 08:48:34.507955 [ 32336 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_140_40 from disk default
2026.05.06 08:48:34.509046 [ 32377 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_110_110_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.509106 [ 32377 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_110_110_0 on disk default
2026.05.06 08:48:34.509181 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_110_110_0
2026.05.06 08:48:34.510163 [ 32377 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_141_41 from disk default
2026.05.06 08:48:34.510590 [ 32396 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_122_122_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.510657 [ 32396 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_122_122_0 on disk default
2026.05.06 08:48:34.510742 [ 32396 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_122_122_0
2026.05.06 08:48:34.510882 [ 32343 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_255_255_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.510927 [ 32343 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_255_255_0 on disk default
2026.05.06 08:48:34.510946 [ 32396 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_109_109_0 from disk default
2026.05.06 08:48:34.510992 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_255_255_0
2026.05.06 08:48:34.511133 [ 32343 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_254_254_0 from disk default
2026.05.06 08:48:34.512370 [ 32347 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_123_148_5 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.512437 [ 32347 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_123_148_5 on disk default
2026.05.06 08:48:34.512592 [ 32347 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_123_148_5
2026.05.06 08:48:34.512884 [ 32338 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_143_39 from disk default
2026.05.06 08:48:34.512889 [ 32394 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_257_257_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.512972 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_257_257_0 on disk default
2026.05.06 08:48:34.513077 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_257_257_0
2026.05.06 08:48:34.513318 [ 32390 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_142_42 from disk default
2026.05.06 08:48:34.514470 [ 32369 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_126_126_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.514545 [ 32369 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_126_126_0 on disk default
2026.05.06 08:48:34.514621 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_126_126_0
2026.05.06 08:48:34.514823 [ 32401 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loading Outdated part 202605_1_144_47 from disk default
2026.05.06 08:48:34.515002 [ 32399 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_111_111_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.515080 [ 32399 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_111_111_0 on disk default
2026.05.06 08:48:34.515150 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_111_111_0
2026.05.06 08:48:34.515273 [ 32399 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_155_68 from disk default
2026.05.06 08:48:34.517393 [ 32344 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_118_118_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.517462 [ 32341 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_126_37 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.517465 [ 32344 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_118_118_0 on disk default
2026.05.06 08:48:34.517545 [ 32341 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_126_37 on disk default
2026.05.06 08:48:34.517633 [ 32341 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_126_37
2026.05.06 08:48:34.517737 [ 32338 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_143_39 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.517779 [ 32341 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loading Outdated part 202605_1_148_40 from disk default
2026.05.06 08:48:34.517800 [ 32338 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_143_39 on disk default
2026.05.06 08:48:34.519105 [ 32392 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_121_121_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.519156 [ 32392 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_121_121_0 on disk default
2026.05.06 08:48:34.520038 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_118_118_0
2026.05.06 08:48:34.520200 [ 32344 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_117_117_0 from disk default
2026.05.06 08:48:34.517061 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_1_131_26 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.520899 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_1_136_27 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.520938 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_1_141_28 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.520971 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_1_146_29 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521002 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_1_151_30 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521033 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_1_156_31 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521066 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_1_161_32 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521096 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_1_166_33 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521128 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_132_132_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521159 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_133_133_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521191 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_134_134_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521222 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_135_135_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521253 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_136_136_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521284 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_137_137_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521314 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_138_138_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521353 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_139_139_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521384 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_140_140_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521415 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_141_141_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521447 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_142_142_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521477 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_143_143_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521520 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_144_144_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521551 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_145_145_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521582 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_146_146_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521614 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_147_147_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521646 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_148_148_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521679 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_149_149_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521712 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_150_150_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521743 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_151_151_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521756 [ 32342 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_111_111_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.521776 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_152_152_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521811 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_153_153_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521843 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_154_154_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521843 [ 32342 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_111_111_0 on disk default
2026.05.06 08:48:34.521874 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_155_155_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521907 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_156_156_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521938 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_157_157_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.521975 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_158_158_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522006 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_159_159_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522037 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_160_160_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522069 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_161_161_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522100 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_162_162_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522106 [ 32359 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_108_108_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.522132 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_163_163_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522165 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_164_164_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522176 [ 32359 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_108_108_0 on disk default
2026.05.06 08:48:34.522197 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_165_165_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522229 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_166_166_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522252 [ 32386 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_107_107_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.522260 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_167_167_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522334 [ 32386 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_107_107_0 on disk default
2026.05.06 08:48:34.522355 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_168_168_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522391 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_169_169_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522422 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_170_170_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522455 [ 32311 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): removePartsFinally: removing 202605_171_171_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.522701 [ 32311 ] {} <Debug> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Removing 48 parts from memory: Parts: [202605_1_131_26, 202605_1_136_27, 202605_1_141_28, 202605_1_146_29, 202605_1_151_30, 202605_1_156_31, 202605_1_161_32, 202605_1_166_33, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0, 202605_167_167_0, 202605_168_168_0, 202605_169_169_0, 202605_170_170_0, 202605_171_171_0]
2026.05.06 08:48:34.519217 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_258_258_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524133 [ 32397 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_109_109_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.524137 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_259_259_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524205 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_261_261_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524226 [ 32397 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_109_109_0 on disk default
2026.05.06 08:48:34.524234 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_262_262_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524281 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_263_263_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524310 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_264_264_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524336 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_265_265_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524363 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_266_266_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524389 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_267_267_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524415 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_268_268_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524442 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_269_269_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524468 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_270_270_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524508 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_271_271_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524534 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_272_272_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524560 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_273_273_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524586 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_274_274_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524612 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_275_275_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524646 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_276_276_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524672 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_277_277_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524697 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_278_278_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524723 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_279_279_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524750 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_280_280_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524775 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_281_281_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524802 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_282_282_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524827 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_283_283_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524853 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_284_284_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524878 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_285_285_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524904 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_286_286_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524929 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_287_287_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524955 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_288_288_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.524980 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_289_289_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525005 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_290_290_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525031 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_291_291_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525056 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_292_292_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525082 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_293_293_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525107 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_294_294_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525132 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_295_295_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525158 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_296_296_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525183 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_297_297_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525212 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_298_298_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525238 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_299_299_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525262 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_300_300_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525288 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_301_301_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525314 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_302_302_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525339 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_303_303_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525365 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_304_304_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525389 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_305_305_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525414 [ 32312 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_306_306_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.525557 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_143_39
2026.05.06 08:48:34.525605 [ 32343 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_254_254_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.525670 [ 32343 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_254_254_0 on disk default
2026.05.06 08:48:34.525730 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_108_108_0 from disk default
2026.05.06 08:48:34.525773 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_254_254_0
2026.05.06 08:48:34.525561 [ 32312 ] {} <Debug> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Removing 48 parts from memory: Parts: [202605_258_258_0, 202605_259_259_0, 202605_261_261_0, 202605_262_262_0, 202605_263_263_0, 202605_264_264_0, 202605_265_265_0, 202605_266_266_0, 202605_267_267_0, 202605_268_268_0, 202605_269_269_0, 202605_270_270_0, 202605_271_271_0, 202605_272_272_0, 202605_273_273_0, 202605_274_274_0, 202605_275_275_0, 202605_276_276_0, 202605_277_277_0, 202605_278_278_0, 202605_279_279_0, 202605_280_280_0, 202605_281_281_0, 202605_282_282_0, 202605_283_283_0, 202605_284_284_0, 202605_285_285_0, 202605_286_286_0, 202605_287_287_0, 202605_288_288_0, 202605_289_289_0, 202605_290_290_0, 202605_291_291_0, 202605_292_292_0, 202605_293_293_0, 202605_294_294_0, 202605_295_295_0, 202605_296_296_0, 202605_297_297_0, 202605_298_298_0, 202605_299_299_0, 202605_300_300_0, 202605_301_301_0, 202605_302_302_0, 202605_303_303_0, 202605_304_304_0, 202605_305_305_0, 202605_306_306_0]
2026.05.06 08:48:34.527407 [ 32385 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_136_39 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.527472 [ 32385 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_136_39 on disk default
2026.05.06 08:48:34.527583 [ 32385 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_136_39
2026.05.06 08:48:34.527690 [ 32378 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_119_119_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.527727 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_121_121_0
2026.05.06 08:48:34.527756 [ 32378 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_119_119_0 on disk default
2026.05.06 08:48:34.527828 [ 32372 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_143_43 from disk default
2026.05.06 08:48:34.527837 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_119_119_0
2026.05.06 08:48:34.527893 [ 32368 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_148_44 from disk default
2026.05.06 08:48:34.528003 [ 32392 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_253_253_0 from disk default
2026.05.06 08:48:34.528068 [ 32351 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_156_69 from disk default
2026.05.06 08:48:34.528739 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_111_111_0
2026.05.06 08:48:34.528883 [ 32342 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_116_116_0 from disk default
2026.05.06 08:48:34.529076 [ 32399 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_155_68 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.529144 [ 32399 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_155_68 on disk default
2026.05.06 08:48:34.529242 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_155_68
2026.05.06 08:48:34.529398 [ 32399 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_107_107_0 from disk default
2026.05.06 08:48:34.530360 [ 32341 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): loadDataPart: inserting 202605_1_148_40 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.530417 [ 32341 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Finished loading Outdated part 202605_1_148_40 on disk default
2026.05.06 08:48:34.530494 [ 32341 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86), part: 202605_1_148_40
2026.05.06 08:48:34.530671 [ 32386 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_107_107_0
2026.05.06 08:48:34.530687 [ 32359 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_108_108_0
2026.05.06 08:48:34.530726 [ 32397 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_109_109_0
2026.05.06 08:48:34.530815 [ 32386 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_152_45 from disk default
2026.05.06 08:48:34.530909 [ 32397 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_153_46 from disk default
2026.05.06 08:48:34.531265 [ 31718 ] {} <Debug> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Loaded 57 outdated data parts asynchronously
2026.05.06 08:48:34.531351 [ 31718 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 1099 ms.
2026.05.06 08:48:34.531358 [ 32359 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_253_258_1 from disk default
2026.05.06 08:48:34.531368 [ 32341 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_154_47 from disk default
2026.05.06 08:48:34.531982 [ 32390 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_142_42 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.532044 [ 32390 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_142_42 on disk default
2026.05.06 08:48:34.532119 [ 32390 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_142_42
2026.05.06 08:48:34.532467 [ 32390 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_1_106_21 from disk default
2026.05.06 08:48:34.532801 [ 32401 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): loadDataPart: inserting 202605_1_144_47 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.532870 [ 32401 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Finished loading Outdated part 202605_1_144_47 on disk default
2026.05.06 08:48:34.535841 [ 32400 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_131_38 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.535915 [ 32400 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_131_38 on disk default
2026.05.06 08:48:34.537297 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383), part: 202605_1_144_47
2026.05.06 08:48:34.537300 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_131_38
2026.05.06 08:48:34.537516 [ 32400 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_253_263_2 from disk default
2026.05.06 08:48:34.537693 [ 32338 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_108_108_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.537754 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_108_108_0 on disk default
2026.05.06 08:48:34.537832 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_108_108_0
2026.05.06 08:48:34.537886 [ 32402 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_120_120_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.537950 [ 32402 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_120_120_0 on disk default
2026.05.06 08:48:34.538011 [ 32394 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_1_111_22 from disk default
2026.05.06 08:48:34.538046 [ 32402 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_120_120_0
2026.05.06 08:48:34.538212 [ 32347 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_157_70 from disk default
2026.05.06 08:48:34.540218 [ 32396 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_109_109_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.540283 [ 32396 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_109_109_0 on disk default
2026.05.06 08:48:34.540365 [ 32396 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_109_109_0
2026.05.06 08:48:34.540549 [ 32369 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_1_116_23 from disk default
2026.05.06 08:48:34.540751 [ 32372 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_143_43 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.540803 [ 32372 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_143_43 on disk default
2026.05.06 08:48:34.540828 [ 32383 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_112_112_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.540884 [ 32383 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_112_112_0 on disk default
2026.05.06 08:48:34.541152 [ 32351 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_156_69 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.541226 [ 32351 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_156_69 on disk default
2026.05.06 08:48:34.541713 [ 31703 ] {} <Debug> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Loaded 50 outdated data parts asynchronously
2026.05.06 08:48:34.541902 [ 32395 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_1_121_24 from disk default
2026.05.06 08:48:34.543073 [ 31703 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 1121 ms.
2026.05.06 08:48:34.543092 [ 32351 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_156_69
2026.05.06 08:48:34.543247 [ 32375 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_1_126_25 from disk default
2026.05.06 08:48:34.545163 [ 32392 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_253_253_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.545261 [ 32392 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_253_253_0 on disk default
2026.05.06 08:48:34.545319 [ 32368 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_148_44 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.545373 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_253_253_0
2026.05.06 08:48:34.545388 [ 32368 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_148_44 on disk default
2026.05.06 08:48:34.545181 [ 32372 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_143_43
2026.05.06 08:48:34.545515 [ 32368 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_148_44
2026.05.06 08:48:34.545564 [ 32392 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_253_268_3 from disk default
2026.05.06 08:48:34.545576 [ 32383 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_112_112_0
2026.05.06 08:48:34.545668 [ 32368 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_115_115_0 from disk default
2026.05.06 08:48:34.545699 [ 32372 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_84_114_6 from disk default
2026.05.06 08:48:34.545767 [ 32385 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_84_119_7 from disk default
2026.05.06 08:48:34.546019 [ 32394 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_1_111_22 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.546071 [ 32394 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_1_111_22 on disk default
2026.05.06 08:48:34.546592 [ 32347 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_157_70 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.546657 [ 32347 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_157_70 on disk default
2026.05.06 08:48:34.548669 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_1_111_22
2026.05.06 08:48:34.548690 [ 32347 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_157_70
2026.05.06 08:48:34.548805 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_253_273_4 from disk default
2026.05.06 08:48:34.549224 [ 32347 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_253_278_5 from disk default
2026.05.06 08:48:34.554710 [ 32375 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_1_126_25 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.554774 [ 32375 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_1_126_25 on disk default
2026.05.06 08:48:34.554797 [ 32395 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_1_121_24 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.554852 [ 32375 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_1_126_25
2026.05.06 08:48:34.554870 [ 32395 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_1_121_24 on disk default
2026.05.06 08:48:34.554949 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_1_121_24
2026.05.06 08:48:34.555094 [ 32395 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loading Outdated part 202605_1_83_15 from disk default
2026.05.06 08:48:34.555236 [ 32375 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_253_283_6 from disk default
2026.05.06 08:48:34.556620 [ 32336 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_140_40 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.556684 [ 32336 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_140_40 on disk default
2026.05.06 08:48:34.556759 [ 32336 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_140_40
2026.05.06 08:48:34.556896 [ 32336 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_1_131_26 from disk default
2026.05.06 08:48:34.557153 [ 32399 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_107_107_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.557204 [ 32399 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_107_107_0 on disk default
2026.05.06 08:48:34.557225 [ 32390 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_1_106_21 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.557272 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_107_107_0
2026.05.06 08:48:34.557286 [ 32390 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_1_106_21 on disk default
2026.05.06 08:48:34.557375 [ 32390 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_1_106_21
2026.05.06 08:48:34.557400 [ 32399 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_253_288_7 from disk default
2026.05.06 08:48:34.557473 [ 32359 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_253_258_1 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.557520 [ 32390 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_155_48 from disk default
2026.05.06 08:48:34.557535 [ 32359 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_253_258_1 on disk default
2026.05.06 08:48:34.557608 [ 32359 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_253_258_1
2026.05.06 08:48:34.557735 [ 32359 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_163_71 from disk default
2026.05.06 08:48:34.560233 [ 32397 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_153_46 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.560287 [ 32397 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_153_46 on disk default
2026.05.06 08:48:34.560352 [ 32397 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_153_46
2026.05.06 08:48:34.560493 [ 32397 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_156_49 from disk default
2026.05.06 08:48:34.560726 [ 32369 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_1_116_23 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.560791 [ 32369 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_1_116_23 on disk default
2026.05.06 08:48:34.560859 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_1_116_23
2026.05.06 08:48:34.560978 [ 32369 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_1_136_27 from disk default
2026.05.06 08:48:34.565904 [ 32392 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_253_268_3 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.565976 [ 32392 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_253_268_3 on disk default
2026.05.06 08:48:34.566093 [ 32392 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_253_268_3
2026.05.06 08:48:34.566360 [ 32396 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_161_50 from disk default
2026.05.06 08:48:34.566395 [ 32394 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_253_273_4 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.566461 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_253_273_4 on disk default
2026.05.06 08:48:34.566569 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_253_273_4
2026.05.06 08:48:34.566686 [ 32347 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_253_278_5 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.566761 [ 32347 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_253_278_5 on disk default
2026.05.06 08:48:34.566821 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loading Outdated part 202605_1_141_28 from disk default
2026.05.06 08:48:34.566858 [ 32347 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_253_278_5
2026.05.06 08:48:34.567774 [ 32347 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_167_72 from disk default
2026.05.06 08:48:34.568273 [ 32375 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_253_283_6 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.568324 [ 32375 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_253_283_6 on disk default
2026.05.06 08:48:34.568389 [ 32375 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_253_283_6
2026.05.06 08:48:34.568406 [ 32377 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_141_41 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.568461 [ 32377 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_141_41 on disk default
2026.05.06 08:48:34.568578 [ 32401 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_168_73 from disk default
2026.05.06 08:48:34.568810 [ 32359 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_163_71 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.568894 [ 32359 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_163_71 on disk default
2026.05.06 08:48:34.572326 [ 32386 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_152_45 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.572402 [ 32386 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_152_45 on disk default
2026.05.06 08:48:34.572436 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_141_41
2026.05.06 08:48:34.572501 [ 32386 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_152_45
2026.05.06 08:48:34.572630 [ 32377 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loading Outdated part 202605_1_165_51 from disk default
2026.05.06 08:48:34.573857 [ 32359 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_163_71
2026.05.06 08:48:34.573865 [ 32349 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loading Outdated part 202605_1_169_74 from disk default
2026.05.06 08:48:34.574040 [ 32361 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_253_294_8 from disk default
2026.05.06 08:48:34.575557 [ 32400 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_253_263_2 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.575652 [ 32400 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_253_263_2 on disk default
2026.05.06 08:48:34.575746 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_253_263_2
2026.05.06 08:48:34.575930 [ 32400 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_253_300_9 from disk default
2026.05.06 08:48:34.576378 [ 32342 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_116_116_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.576450 [ 32369 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_1_136_27 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.576535 [ 32369 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_1_136_27 on disk default
2026.05.06 08:48:34.576621 [ 32369 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_1_136_27
2026.05.06 08:48:34.578675 [ 32372 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_84_114_6 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.578724 [ 32372 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_84_114_6 on disk default
2026.05.06 08:48:34.576451 [ 32342 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_116_116_0 on disk default
2026.05.06 08:48:34.579305 [ 32349 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_169_74 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.579360 [ 32349 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_169_74 on disk default
2026.05.06 08:48:34.579444 [ 32401 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_168_73 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.579524 [ 32401 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_168_73 on disk default
2026.05.06 08:48:34.579704 [ 32343 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_252_252_0 from disk default
2026.05.06 08:48:34.579859 [ 32372 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_84_114_6
2026.05.06 08:48:34.579977 [ 32372 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_251_251_0 from disk default
2026.05.06 08:48:34.580541 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_116_116_0
2026.05.06 08:48:34.580677 [ 32342 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_250_250_0 from disk default
2026.05.06 08:48:34.580744 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_169_74
2026.05.06 08:48:34.580770 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_168_73
2026.05.06 08:48:34.580861 [ 32349 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_249_249_0 from disk default
2026.05.06 08:48:34.580904 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_248_248_0 from disk default
2026.05.06 08:48:34.582034 [ 32399 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_253_288_7 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.582100 [ 32399 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_253_288_7 on disk default
2026.05.06 08:48:34.583122 [ 32341 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_154_47 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.583178 [ 32341 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_154_47 on disk default
2026.05.06 08:48:34.584545 [ 32361 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_253_294_8 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.584611 [ 32361 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_253_294_8 on disk default
2026.05.06 08:48:34.586004 [ 32367 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_105_105_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.586064 [ 32367 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_105_105_0 on disk default
2026.05.06 08:48:34.586664 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_253_288_7
2026.05.06 08:48:34.586788 [ 32399 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_247_247_0 from disk default
2026.05.06 08:48:34.588839 [ 32338 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_1_141_28 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.588900 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_1_141_28 on disk default
2026.05.06 08:48:34.589108 [ 32341 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_154_47
2026.05.06 08:48:34.589125 [ 32361 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_253_294_8
2026.05.06 08:48:34.589237 [ 32341 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_246_246_0 from disk default
2026.05.06 08:48:34.589309 [ 32370 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_245_245_0 from disk default
2026.05.06 08:48:34.590315 [ 32343 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_252_252_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.590363 [ 32343 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_252_252_0 on disk default
2026.05.06 08:48:34.590506 [ 32367 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_105_105_0
2026.05.06 08:48:34.590507 [ 32385 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_84_119_7 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.590569 [ 32338 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_1_141_28
2026.05.06 08:48:34.590624 [ 32385 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_84_119_7 on disk default
2026.05.06 08:48:34.590678 [ 32360 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_244_244_0 from disk default
2026.05.06 08:48:34.590701 [ 32385 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_84_119_7
2026.05.06 08:48:34.590881 [ 32404 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_243_243_0 from disk default
2026.05.06 08:48:34.590996 [ 32373 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_242_242_0 from disk default
2026.05.06 08:48:34.591122 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_252_252_0
2026.05.06 08:48:34.591264 [ 32343 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_241_241_0 from disk default
2026.05.06 08:48:34.592181 [ 32372 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_251_251_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.592238 [ 32372 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_251_251_0 on disk default
2026.05.06 08:48:34.592309 [ 32372 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_251_251_0
2026.05.06 08:48:34.592457 [ 32400 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_253_300_9 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.592466 [ 32402 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_240_240_0 from disk default
2026.05.06 08:48:34.592533 [ 32400 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_253_300_9 on disk default
2026.05.06 08:48:34.592625 [ 32400 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_253_300_9
2026.05.06 08:48:34.594775 [ 32347 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): loadDataPart: inserting 202605_1_167_72 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.594839 [ 32347 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Finished loading Outdated part 202605_1_167_72 on disk default
2026.05.06 08:48:34.594928 [ 32347 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa), part: 202605_1_167_72
2026.05.06 08:48:34.595541 [ 32368 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_115_115_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.595597 [ 32368 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_115_115_0 on disk default
2026.05.06 08:48:34.595682 [ 32368 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_115_115_0
2026.05.06 08:48:34.595750 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_239_239_0 from disk default
2026.05.06 08:48:34.595837 [ 32368 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_238_238_0 from disk default
2026.05.06 08:48:34.597832 [ 32336 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): loadDataPart: inserting 202605_1_131_26 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.597885 [ 32336 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Finished loading Outdated part 202605_1_131_26 on disk default
2026.05.06 08:48:34.598904 [ 32336 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040), part: 202605_1_131_26
2026.05.06 08:48:34.599758 [ 32349 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_249_249_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.599812 [ 32349 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_249_249_0 on disk default
2026.05.06 08:48:34.599883 [ 32349 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_249_249_0
2026.05.06 08:48:34.600034 [ 32351 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_237_237_0 from disk default
2026.05.06 08:48:34.600128 [ 32343 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_241_241_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.600177 [ 32343 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_241_241_0 on disk default
2026.05.06 08:48:34.600242 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_241_241_0
2026.05.06 08:48:34.600329 [ 32343 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_236_236_0 from disk default
2026.05.06 08:48:34.601819 [ 32344 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_117_117_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.601913 [ 32344 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_117_117_0 on disk default
2026.05.06 08:48:34.602004 [ 32397 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_156_49 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.602006 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_117_117_0
2026.05.06 08:48:34.602081 [ 32397 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_156_49 on disk default
2026.05.06 08:48:34.602164 [ 32397 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_156_49
2026.05.06 08:48:34.602167 [ 32344 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_235_235_0 from disk default
2026.05.06 08:48:34.602265 [ 32397 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_234_234_0 from disk default
2026.05.06 08:48:34.603372 [ 32390 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_155_48 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.603430 [ 32390 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_155_48 on disk default
2026.05.06 08:48:34.603820 [ 31701 ] {} <Debug> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Loaded 61 outdated data parts asynchronously
2026.05.06 08:48:34.603967 [ 32382 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_233_233_0 from disk default
2026.05.06 08:48:34.606008 [ 32336 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_222_232_2 from disk default
2026.05.06 08:48:34.606079 [ 31698 ] {} <Debug> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Loaded 48 outdated data parts asynchronously
2026.05.06 08:48:34.606846 [ 32404 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_243_243_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.606900 [ 32404 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_243_243_0 on disk default
2026.05.06 08:48:34.608617 [ 31701 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 1187 ms.
2026.05.06 08:48:34.608781 [ 31698 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 1188 ms.
2026.05.06 08:48:34.608839 [ 32404 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_243_243_0
2026.05.06 08:48:34.608878 [ 32390 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_155_48
2026.05.06 08:48:34.609008 [ 32378 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_222_237_3 from disk default
2026.05.06 08:48:34.609251 [ 32402 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_240_240_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.609313 [ 32402 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_240_240_0 on disk default
2026.05.06 08:48:34.609348 [ 32373 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_242_242_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.609404 [ 32373 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_242_242_0 on disk default
2026.05.06 08:48:34.609409 [ 32402 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_240_240_0
2026.05.06 08:48:34.609551 [ 32390 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_222_242_4 from disk default
2026.05.06 08:48:34.609613 [ 32377 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_165_51 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.609627 [ 32402 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_222_247_5 from disk default
2026.05.06 08:48:34.609649 [ 32360 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_244_244_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.609681 [ 32377 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_165_51 on disk default
2026.05.06 08:48:34.609728 [ 32360 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_244_244_0 on disk default
2026.05.06 08:48:34.609742 [ 32394 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_239_239_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.609801 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_239_239_0 on disk default
2026.05.06 08:48:34.610290 [ 32399 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_247_247_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.610349 [ 32399 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_247_247_0 on disk default
2026.05.06 08:48:34.610718 [ 32341 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_246_246_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.610768 [ 32341 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_246_246_0 on disk default
2026.05.06 08:48:34.611397 [ 32373 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_242_242_0
2026.05.06 08:48:34.611398 [ 32317 ] {} <Trace> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Found 57 old parts to remove. Parts: [202605_1_112_25, 202605_1_117_26, 202605_1_122_27, 202605_1_124_28, 202605_1_125_29, 202605_1_126_30, 202605_1_127_31, 202605_1_128_32, 202605_1_129_33, 202605_1_130_34, 202605_1_131_35, 202605_1_132_36, 202605_1_133_37, 202605_1_138_38, 202605_1_143_39, 202605_1_148_40, 202605_113_113_0, 202605_114_114_0, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0]
2026.05.06 08:48:34.611474 [ 32396 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): loadDataPart: inserting 202605_1_161_50 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.611558 [ 32396 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Finished loading Outdated part 202605_1_161_50 on disk default
2026.05.06 08:48:34.611597 [ 32398 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_222_252_6 from disk default
2026.05.06 08:48:34.611702 [ 32360 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_244_244_0
2026.05.06 08:48:34.611703 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_165_51
2026.05.06 08:48:34.611861 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_239_239_0
2026.05.06 08:48:34.611909 [ 32377 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_131_221_14 from disk default
2026.05.06 08:48:34.611915 [ 32399 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_247_247_0
2026.05.06 08:48:34.611933 [ 32376 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_131_252_15 from disk default
2026.05.06 08:48:34.612005 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loading Outdated part 202605_1_130_13 from disk default
2026.05.06 08:48:34.612578 [ 32342 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_250_250_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.612636 [ 32342 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_250_250_0 on disk default
2026.05.06 08:48:34.612651 [ 32370 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_245_245_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.612703 [ 32370 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_245_245_0 on disk default
2026.05.06 08:48:34.613057 [ 32396 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185), part: 202605_1_161_50
2026.05.06 08:48:34.613141 [ 32317 ] {} <Debug> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Removing 57 parts from filesystem (serially): Parts: [202605_1_112_25, 202605_1_117_26, 202605_1_122_27, 202605_1_124_28, 202605_1_125_29, 202605_1_126_30, 202605_1_127_31, 202605_1_128_32, 202605_1_129_33, 202605_1_130_34, 202605_1_131_35, 202605_1_132_36, 202605_1_133_37, 202605_1_138_38, 202605_1_143_39, 202605_1_148_40, 202605_113_113_0, 202605_114_114_0, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0]
2026.05.06 08:48:34.613507 [ 31711 ] {} <Debug> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Loaded 57 outdated data parts asynchronously
2026.05.06 08:48:34.614471 [ 32341 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_246_246_0
2026.05.06 08:48:34.616084 [ 32351 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_237_237_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.616160 [ 32351 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_237_237_0 on disk default
2026.05.06 08:48:34.616248 [ 32351 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_237_237_0
2026.05.06 08:48:34.616635 [ 32343 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_236_236_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.616707 [ 32343 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_236_236_0 on disk default
2026.05.06 08:48:34.616719 [ 32401 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_248_248_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.616774 [ 32401 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_248_248_0 on disk default
2026.05.06 08:48:34.616798 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_236_236_0
2026.05.06 08:48:34.616854 [ 32401 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_248_248_0
2026.05.06 08:48:34.616855 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_245_245_0
2026.05.06 08:48:34.617219 [ 32382 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_233_233_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.617265 [ 32382 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_233_233_0 on disk default
2026.05.06 08:48:34.617471 [ 31711 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 1194 ms.
2026.05.06 08:48:34.617649 [ 32342 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_250_250_0
2026.05.06 08:48:34.617698 [ 32344 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_235_235_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.617749 [ 32344 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_235_235_0 on disk default
2026.05.06 08:48:34.617817 [ 32344 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_235_235_0
2026.05.06 08:48:34.618839 [ 32336 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_222_232_2 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.618884 [ 32336 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_222_232_2 on disk default
2026.05.06 08:48:34.618950 [ 32336 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_222_232_2
2026.05.06 08:48:34.619010 [ 32382 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_233_233_0
2026.05.06 08:48:34.619389 [ 32398 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_222_252_6 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.619442 [ 32398 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_222_252_6 on disk default
2026.05.06 08:48:34.619520 [ 32398 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_222_252_6
2026.05.06 08:48:34.619568 [ 32402 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_222_247_5 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.619683 [ 32402 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_222_247_5 on disk default
2026.05.06 08:48:34.619781 [ 32402 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_222_247_5
2026.05.06 08:48:34.621774 [ 32390 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_222_242_4 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.621824 [ 32390 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_222_242_4 on disk default
2026.05.06 08:48:34.621887 [ 32390 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_222_242_4
2026.05.06 08:48:34.621887 [ 32378 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_222_237_3 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.621953 [ 32378 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_222_237_3 on disk default
2026.05.06 08:48:34.622031 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_222_237_3
2026.05.06 08:48:34.622529 [ 32397 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_234_234_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.622609 [ 32397 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_234_234_0 on disk default
2026.05.06 08:48:34.622765 [ 32397 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_234_234_0
2026.05.06 08:48:34.627088 [ 32371 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_106_106_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.627159 [ 32371 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_106_106_0 on disk default
2026.05.06 08:48:34.627492 [ 32371 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_106_106_0
2026.05.06 08:48:34.628663 [ 32376 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_131_252_15 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.628721 [ 32376 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_131_252_15 on disk default
2026.05.06 08:48:34.628791 [ 32376 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_131_252_15
2026.05.06 08:48:34.628875 [ 32368 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_238_238_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.628941 [ 32368 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_238_238_0 on disk default
2026.05.06 08:48:34.629012 [ 32368 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_238_238_0
2026.05.06 08:48:34.629128 [ 32394 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_1_130_13 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.629181 [ 32394 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_1_130_13 on disk default
2026.05.06 08:48:34.629251 [ 32394 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_1_130_13
2026.05.06 08:48:34.632697 [ 32377 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): loadDataPart: inserting 202605_131_221_14 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.632752 [ 32377 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Finished loading Outdated part 202605_131_221_14 on disk default
2026.05.06 08:48:34.632821 [ 32377 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d), part: 202605_131_221_14
2026.05.06 08:48:34.633088 [ 31710 ] {} <Debug> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Loaded 91 outdated data parts asynchronously
2026.05.06 08:48:34.633175 [ 31710 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 1208 ms.
2026.05.06 08:48:34.638880 [ 32337 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_1_101_20 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.638942 [ 32337 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_1_101_20 on disk default
2026.05.06 08:48:34.639011 [ 32337 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_1_101_20
2026.05.06 08:48:34.641214 [ 32388 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_110_110_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.641333 [ 32388 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_110_110_0 on disk default
2026.05.06 08:48:34.641454 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_110_110_0
2026.05.06 08:48:34.644734 [ 32381 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_1_106_21 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.644826 [ 32381 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_1_106_21 on disk default
2026.05.06 08:48:34.644972 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_1_106_21
2026.05.06 08:48:34.645540 [ 32395 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): loadDataPart: inserting 202605_1_83_15 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.645568 [ 32387 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_102_102_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.645584 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_112_25 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645631 [ 32387 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_102_102_0 on disk default
2026.05.06 08:48:34.645653 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_117_26 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645607 [ 32395 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Finished loading Outdated part 202605_1_83_15 on disk default
2026.05.06 08:48:34.645689 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_122_27 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645717 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_124_28 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645732 [ 32387 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_102_102_0
2026.05.06 08:48:34.645745 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_125_29 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645757 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad), part: 202605_1_83_15
2026.05.06 08:48:34.645771 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_126_30 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645799 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_127_31 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645825 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_128_32 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645851 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_129_33 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645876 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_130_34 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645903 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_131_35 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645928 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_132_36 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645918 [ 31720 ] {} <Debug> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Loaded 48 outdated data parts asynchronously
2026.05.06 08:48:34.645963 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_133_37 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.645990 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_138_38 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646015 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_143_39 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646021 [ 31720 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 1204 ms.
2026.05.06 08:48:34.646041 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_1_148_40 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646073 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_113_113_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646098 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_114_114_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646123 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_115_115_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646149 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_116_116_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646175 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_117_117_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646200 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_118_118_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646226 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_119_119_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646252 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_120_120_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646277 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_121_121_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646303 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_122_122_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646328 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_123_123_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646354 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_124_124_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646380 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_125_125_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646405 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_126_126_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646430 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_127_127_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646456 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_128_128_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646490 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_129_129_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646515 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_130_130_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646540 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_131_131_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646565 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_132_132_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646594 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_133_133_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646619 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_134_134_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646644 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_135_135_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646669 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_136_136_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646694 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_137_137_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646719 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_138_138_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646744 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_139_139_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646769 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_140_140_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646794 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_141_141_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646818 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_142_142_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646843 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_143_143_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646867 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_144_144_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646892 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_145_145_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646917 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_146_146_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646942 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_147_147_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646966 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_148_148_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.646991 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_149_149_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.647016 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_150_150_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.647040 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_151_151_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.647064 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_152_152_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.647091 [ 32317 ] {} <Test> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): removePartsFinally: removing 202605_153_153_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.647219 [ 32317 ] {} <Debug> system.opentelemetry_span_log (c1ffe847-07d0-4c3e-b0d1-b1f446cadf86): Removing 57 parts from memory: Parts: [202605_1_112_25, 202605_1_117_26, 202605_1_122_27, 202605_1_124_28, 202605_1_125_29, 202605_1_126_30, 202605_1_127_31, 202605_1_128_32, 202605_1_129_33, 202605_1_130_34, 202605_1_131_35, 202605_1_132_36, 202605_1_133_37, 202605_1_138_38, 202605_1_143_39, 202605_1_148_40, 202605_113_113_0, 202605_114_114_0, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0]
2026.05.06 08:48:34.650972 [ 32374 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_104_104_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.650988 [ 31633 ] {} <Debug> SystemLog: Creating system.error_log from error_log
2026.05.06 08:48:34.651057 [ 32374 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_104_104_0 on disk default
2026.05.06 08:48:34.651154 [ 32374 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_104_104_0
2026.05.06 08:48:34.652921 [ 32365 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_1_116_23 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.653011 [ 32365 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_1_116_23 on disk default
2026.05.06 08:48:34.653124 [ 32365 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_1_116_23
2026.05.06 08:48:34.654197 [ 32403 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_103_103_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.654276 [ 32403 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_103_103_0 on disk default
2026.05.06 08:48:34.654367 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_103_103_0
2026.05.06 08:48:34.656142 [ 32340 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_1_111_22 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.656202 [ 32340 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_1_111_22 on disk default
2026.05.06 08:48:34.656310 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_1_111_22
2026.05.06 08:48:34.656861 [ 32339 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_1_121_24 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.656936 [ 32339 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_1_121_24 on disk default
2026.05.06 08:48:34.657001 [ 32339 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_1_121_24
2026.05.06 08:48:34.658615 [ 31633 ] {} <Debug> SystemLog: Creating system.filesystem_cache_log from filesystem_cache_log
2026.05.06 08:48:34.659747 [ 32353 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): loadDataPart: inserting 202605_1_126_25 (state Outdated) into data_parts_indexes
2026.05.06 08:48:34.659802 [ 32353 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Finished loading Outdated part 202605_1_126_25 on disk default
2026.05.06 08:48:34.659872 [ 32353 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_1_126_25
2026.05.06 08:48:34.660214 [ 31687 ] {} <Debug> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Loaded 36 outdated data parts asynchronously
2026.05.06 08:48:34.660395 [ 31687 ] {} <Trace> MergeTreeData::loadOutdatedDataParts: Execution took 1240 ms.
2026.05.06 08:48:34.665456 [ 31633 ] {} <Debug> SystemLog: Not creating system.filesystem_read_prefetches_log since corresponding section 'filesystem_read_prefetches_log' is missing from config
2026.05.06 08:48:34.665572 [ 31633 ] {} <Debug> SystemLog: Creating system.s3queue_log from s3queue_log
2026.05.06 08:48:34.671591 [ 31633 ] {} <Debug> SystemLog: Not creating system.azure_queue_log since corresponding section 'azure_queue_log' is missing from config
2026.05.06 08:48:34.671722 [ 31633 ] {} <Debug> SystemLog: Creating system.asynchronous_metric_log from asynchronous_metric_log
2026.05.06 08:48:34.677968 [ 31633 ] {} <Debug> SystemLog: Creating system.opentelemetry_span_log from opentelemetry_span_log
2026.05.06 08:48:34.685376 [ 31633 ] {} <Debug> SystemLog: Creating system.query_views_log from query_views_log
2026.05.06 08:48:34.691563 [ 32315 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Found 48 old parts to remove. Parts: [202605_1_83_15, 202605_84_114_6, 202605_84_119_7, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_123_128_1, 202605_123_133_2, 202605_123_138_3, 202605_123_143_4, 202605_123_148_5, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0]
2026.05.06 08:48:34.691859 [ 32315 ] {} <Debug> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Removing 48 parts from filesystem (serially): Parts: [202605_1_83_15, 202605_84_114_6, 202605_84_119_7, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_123_128_1, 202605_123_133_2, 202605_123_138_3, 202605_123_143_4, 202605_123_148_5, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0]
2026.05.06 08:48:34.694692 [ 31633 ] {} <Debug> SystemLog: Creating system.zookeeper_log from zookeeper_log
2026.05.06 08:48:34.709371 [ 31633 ] {} <Debug> SystemLog: Creating system.session_log from session_log
2026.05.06 08:48:34.723691 [ 31633 ] {} <Debug> SystemLog: Creating system.transactions_info_log from transactions_info_log
2026.05.06 08:48:34.730395 [ 31633 ] {} <Debug> SystemLog: Creating system.processors_profile_log from processors_profile_log
2026.05.06 08:48:34.731369 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_1_83_15 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731432 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_84_114_6 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731460 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_84_119_7 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731495 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_115_115_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731522 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_116_116_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731548 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_117_117_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731575 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_118_118_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731613 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_119_119_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731641 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_120_120_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731666 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_121_121_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731692 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_122_122_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731717 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_123_123_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731743 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_123_128_1 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731773 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_123_133_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731798 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_123_138_3 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731824 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_123_143_4 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731850 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_123_148_5 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731876 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_124_124_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731901 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_125_125_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731935 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_126_126_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731961 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_127_127_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.731986 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_128_128_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732011 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_129_129_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732036 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_130_130_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732061 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_131_131_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732091 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_132_132_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732116 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_133_133_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732142 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_134_134_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732167 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_135_135_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732192 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_136_136_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732217 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_137_137_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732243 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_138_138_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732268 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_139_139_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732292 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_140_140_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732318 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_141_141_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732343 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_142_142_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732368 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_143_143_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732392 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_144_144_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732417 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_145_145_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732441 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_146_146_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732470 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_147_147_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732502 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_148_148_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732527 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_149_149_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732551 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_150_150_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732575 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_151_151_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732600 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_152_152_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732624 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_153_153_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732648 [ 32315 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): removePartsFinally: removing 202605_154_154_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.732783 [ 32315 ] {} <Debug> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Removing 48 parts from memory: Parts: [202605_1_83_15, 202605_84_114_6, 202605_84_119_7, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_123_128_1, 202605_123_133_2, 202605_123_138_3, 202605_123_143_4, 202605_123_148_5, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0]
2026.05.06 08:48:34.737037 [ 31633 ] {} <Debug> SystemLog: Creating system.asynchronous_insert_log from asynchronous_insert_log
2026.05.06 08:48:34.743978 [ 32318 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Found 6 old parts to remove. Parts: [202605_1_15_3, 202605_16_16_0, 202605_17_17_0, 202605_18_18_0, 202605_19_19_0, 202605_20_20_0]
2026.05.06 08:48:34.744270 [ 32318 ] {} <Debug> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Removing 6 parts from filesystem (serially): Parts: [202605_1_15_3, 202605_16_16_0, 202605_17_17_0, 202605_18_18_0, 202605_19_19_0, 202605_20_20_0]
2026.05.06 08:48:34.745720 [ 31633 ] {} <Debug> SystemLog: Creating system.backup_log from backup_log
2026.05.06 08:48:34.749883 [ 32318 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): removePartsFinally: removing 202605_1_15_3 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.749981 [ 32318 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): removePartsFinally: removing 202605_16_16_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.750040 [ 32318 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): removePartsFinally: removing 202605_17_17_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.750089 [ 32318 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): removePartsFinally: removing 202605_18_18_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.750138 [ 32318 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): removePartsFinally: removing 202605_19_19_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.750200 [ 32318 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): removePartsFinally: removing 202605_20_20_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.750267 [ 32318 ] {} <Debug> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Removing 6 parts from memory: Parts: [202605_1_15_3, 202605_16_16_0, 202605_17_17_0, 202605_18_18_0, 202605_19_19_0, 202605_20_20_0]
2026.05.06 08:48:34.753393 [ 31633 ] {} <Debug> SystemLog: Creating system.blob_storage_log from blob_storage_log
2026.05.06 08:48:34.761571 [ 31633 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2026.05.06 08:48:34.763553 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.763641 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.764173 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.764233 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.764738 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.764841 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.765340 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.765403 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.765895 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.765961 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.766447 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.766521 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.766979 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.767040 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.768148 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.768223 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.770683 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.770749 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.771628 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.771714 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.773309 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.773443 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.774703 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.774763 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.775802 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.775879 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.776805 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.776870 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.777997 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.778083 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.778905 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.778970 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.779979 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.780040 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.780936 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.781002 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.781570 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.781627 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.782210 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.782290 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.782883 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.782941 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.783437 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.783503 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.784040 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.784112 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.784705 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.784765 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.785230 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.785289 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.785999 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.786060 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.786588 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.786650 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.786835 [ 32318 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Found 48 old parts to remove. Parts: [202605_1_106_21, 202605_1_111_22, 202605_1_116_23, 202605_1_121_24, 202605_1_126_25, 202605_1_131_26, 202605_1_136_27, 202605_1_141_28, 202605_107_107_0, 202605_108_108_0, 202605_109_109_0, 202605_110_110_0, 202605_111_111_0, 202605_112_112_0, 202605_113_113_0, 202605_114_114_0, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0]
2026.05.06 08:48:34.786994 [ 32318 ] {} <Debug> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Removing 48 parts from filesystem (serially): Parts: [202605_1_106_21, 202605_1_111_22, 202605_1_116_23, 202605_1_121_24, 202605_1_126_25, 202605_1_131_26, 202605_1_136_27, 202605_1_141_28, 202605_107_107_0, 202605_108_108_0, 202605_109_109_0, 202605_110_110_0, 202605_111_111_0, 202605_112_112_0, 202605_113_113_0, 202605_114_114_0, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0]
2026.05.06 08:48:34.787957 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.788025 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.788673 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.788730 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.791175 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.791237 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.792056 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.792116 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.792694 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.792749 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.793289 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.793360 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.794106 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.794168 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.795353 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.795418 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.796697 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.796792 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.798218 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.798283 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.799508 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.799591 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.801991 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.802055 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.804235 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.804299 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.805108 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.805181 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.806533 [ 32312 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Found 57 old parts to remove. Parts: [202605_1_125_36, 202605_1_126_37, 202605_1_131_38, 202605_1_136_39, 202605_1_140_40, 202605_1_141_41, 202605_1_142_42, 202605_1_143_43, 202605_1_148_44, 202605_1_152_45, 202605_1_153_46, 202605_1_154_47, 202605_1_155_48, 202605_1_156_49, 202605_1_161_50, 202605_1_165_51, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0]
2026.05.06 08:48:34.806693 [ 32312 ] {} <Debug> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Removing 57 parts from filesystem (serially): Parts: [202605_1_125_36, 202605_1_126_37, 202605_1_131_38, 202605_1_136_39, 202605_1_140_40, 202605_1_141_41, 202605_1_142_42, 202605_1_143_43, 202605_1_148_44, 202605_1_152_45, 202605_1_153_46, 202605_1_154_47, 202605_1_155_48, 202605_1_156_49, 202605_1_161_50, 202605_1_165_51, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0]
2026.05.06 08:48:34.806767 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.806828 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.807630 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.807696 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.808234 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.808291 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.808967 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.809044 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.809680 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.809739 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.810215 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.810292 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.811463 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.811537 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.812430 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.812497 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.813184 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.813281 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.817339 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.817423 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.818836 [ 32315 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Found 54 old parts to remove. Parts: [202605_1_136_27, 202605_1_141_28, 202605_1_146_29, 202605_1_151_30, 202605_1_156_31, 202605_1_161_32, 202605_1_166_33, 202605_1_171_34, 202605_1_176_35, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0, 202605_167_167_0, 202605_168_168_0, 202605_169_169_0, 202605_170_170_0, 202605_171_171_0, 202605_172_172_0, 202605_173_173_0, 202605_174_174_0, 202605_175_175_0, 202605_176_176_0, 202605_177_177_0, 202605_178_178_0, 202605_179_179_0, 202605_180_180_0, 202605_181_181_0]
2026.05.06 08:48:34.819080 [ 32315 ] {} <Debug> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Removing 54 parts from filesystem (serially): Parts: [202605_1_136_27, 202605_1_141_28, 202605_1_146_29, 202605_1_151_30, 202605_1_156_31, 202605_1_161_32, 202605_1_166_33, 202605_1_171_34, 202605_1_176_35, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0, 202605_167_167_0, 202605_168_168_0, 202605_169_169_0, 202605_170_170_0, 202605_171_171_0, 202605_172_172_0, 202605_173_173_0, 202605_174_174_0, 202605_175_175_0, 202605_176_176_0, 202605_177_177_0, 202605_178_178_0, 202605_179_179_0, 202605_180_180_0, 202605_181_181_0]
2026.05.06 08:48:34.819593 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.819694 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.820638 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.820763 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.821544 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.821672 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.823062 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.823137 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.823790 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_1_106_21 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.823845 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_1_111_22 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.823872 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_1_116_23 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.823898 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_1_121_24 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.823926 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_1_126_25 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.823956 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_1_131_26 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.823986 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_1_136_27 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824012 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_1_141_28 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824039 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_107_107_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824065 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_108_108_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824057 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.824093 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_109_109_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824120 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_110_110_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824146 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_111_111_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824144 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.824174 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_112_112_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824200 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_113_113_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824226 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_114_114_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824258 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_115_115_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824284 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_116_116_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824310 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_117_117_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824337 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_118_118_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824362 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_119_119_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824388 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_120_120_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824416 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_121_121_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824442 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_122_122_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824466 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_123_123_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824499 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_124_124_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824525 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_125_125_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824551 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_126_126_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824577 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_127_127_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824603 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_128_128_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824628 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_129_129_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824654 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_130_130_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824680 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_131_131_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824705 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_132_132_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824730 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_133_133_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824755 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_134_134_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824771 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.824781 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_135_135_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824825 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_136_136_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824838 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.824849 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_137_137_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824874 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_138_138_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824899 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_139_139_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824925 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_140_140_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824953 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_141_141_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.824978 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_142_142_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.825003 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_143_143_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.825028 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_144_144_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.825052 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_145_145_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.825082 [ 32318 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): removePartsFinally: removing 202605_146_146_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.825196 [ 32318 ] {} <Debug> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Removing 48 parts from memory: Parts: [202605_1_106_21, 202605_1_111_22, 202605_1_116_23, 202605_1_121_24, 202605_1_126_25, 202605_1_131_26, 202605_1_136_27, 202605_1_141_28, 202605_107_107_0, 202605_108_108_0, 202605_109_109_0, 202605_110_110_0, 202605_111_111_0, 202605_112_112_0, 202605_113_113_0, 202605_114_114_0, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0]
2026.05.06 08:48:34.825877 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.825955 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.826611 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.826671 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.830364 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.830447 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.834120 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.834211 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.835690 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.835762 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.838859 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.838929 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.841938 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.842009 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.843669 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.843739 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.844904 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.844974 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.849295 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.849387 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.850136 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_1_136_27 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850206 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_1_141_28 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850235 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_1_146_29 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850262 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_1_151_30 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850289 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_1_156_31 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850317 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_1_161_32 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850344 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_1_166_33 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850371 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_1_171_34 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850396 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_1_176_35 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850423 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_137_137_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850457 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_138_138_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850493 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_139_139_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850529 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_140_140_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850556 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_141_141_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850583 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_142_142_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850609 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_143_143_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850635 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_144_144_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850661 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_145_145_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850687 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_146_146_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850714 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_147_147_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850740 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_148_148_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850766 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_149_149_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850792 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_150_150_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850818 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_151_151_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850845 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_152_152_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850870 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_153_153_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850896 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_154_154_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850922 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_155_155_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850953 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_156_156_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.850958 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.850979 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_157_157_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851010 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_158_158_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851041 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_159_159_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851041 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.851067 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_160_160_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851098 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_161_161_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851124 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_162_162_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851150 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_163_163_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851175 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_164_164_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851201 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_165_165_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851226 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_166_166_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851253 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_167_167_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851278 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_168_168_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851303 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_169_169_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851329 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_170_170_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851355 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_171_171_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851389 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_172_172_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851415 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_173_173_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851440 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_174_174_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851465 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_175_175_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851502 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_176_176_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851528 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_177_177_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851553 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_178_178_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851583 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_179_179_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851618 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_180_180_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851644 [ 32315 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): removePartsFinally: removing 202605_181_181_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.851770 [ 32315 ] {} <Debug> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Removing 54 parts from memory: Parts: [202605_1_136_27, 202605_1_141_28, 202605_1_146_29, 202605_1_151_30, 202605_1_156_31, 202605_1_161_32, 202605_1_166_33, 202605_1_171_34, 202605_1_176_35, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0, 202605_167_167_0, 202605_168_168_0, 202605_169_169_0, 202605_170_170_0, 202605_171_171_0, 202605_172_172_0, 202605_173_173_0, 202605_174_174_0, 202605_175_175_0, 202605_176_176_0, 202605_177_177_0, 202605_178_178_0, 202605_179_179_0, 202605_180_180_0, 202605_181_181_0]
2026.05.06 08:48:34.852419 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.852504 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.853366 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.853431 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.854576 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.854640 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.858347 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.858420 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.860054 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.860160 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.862014 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.862103 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.863340 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.863405 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.864987 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.865064 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.865829 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.865895 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.866933 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_125_36 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.866982 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_126_37 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867010 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_131_38 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867038 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_136_39 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867065 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_140_40 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867091 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_141_41 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867117 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_142_42 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867145 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_143_43 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867171 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_148_44 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867202 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_152_45 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867228 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_153_46 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867255 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_154_47 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867281 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_155_48 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867307 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_156_49 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867333 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_161_50 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867359 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_1_165_51 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867386 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_126_126_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867413 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_127_127_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867439 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_128_128_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867466 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_129_129_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867500 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_130_130_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867533 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_131_131_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867558 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_132_132_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867585 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_133_133_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867589 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.867622 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_134_134_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867651 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_135_135_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867676 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_136_136_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867680 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.867702 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_137_137_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867727 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_138_138_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867754 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_139_139_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867779 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_140_140_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867808 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_141_141_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867838 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_142_142_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867864 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_143_143_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867889 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_144_144_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867917 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_145_145_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867943 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_146_146_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867968 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_147_147_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.867994 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_148_148_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868019 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_149_149_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868044 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_150_150_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868074 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_151_151_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868099 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_152_152_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868125 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_153_153_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868149 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_154_154_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868175 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_155_155_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868200 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_156_156_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868225 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_157_157_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868250 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_158_158_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868275 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_159_159_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868300 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_160_160_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868326 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_161_161_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868350 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_162_162_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868375 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_163_163_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868400 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_164_164_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868425 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_165_165_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868449 [ 32312 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): removePartsFinally: removing 202605_166_166_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.868592 [ 32312 ] {} <Debug> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Removing 57 parts from memory: Parts: [202605_1_125_36, 202605_1_126_37, 202605_1_131_38, 202605_1_136_39, 202605_1_140_40, 202605_1_141_41, 202605_1_142_42, 202605_1_143_43, 202605_1_148_44, 202605_1_152_45, 202605_1_153_46, 202605_1_154_47, 202605_1_155_48, 202605_1_156_49, 202605_1_161_50, 202605_1_165_51, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0]
2026.05.06 08:48:34.868781 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.868849 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.869523 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.869586 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.870603 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.870673 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.871702 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.871831 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.872811 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.872872 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.873961 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.874042 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.874993 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.875061 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.875944 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.876011 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.876912 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.877005 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.878128 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.878193 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.879155 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.879227 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.881037 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.881115 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.883842 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.883928 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.884771 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.884867 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.885795 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.885862 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.886474 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.886572 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.887965 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.888034 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.889181 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.889262 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.890433 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.890519 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.891559 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:34.891633 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:34.913295 [ 32316 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Found 50 old parts to remove. Parts: [202605_1_116_33, 202605_1_121_34, 202605_1_122_35, 202605_1_123_36, 202605_1_124_37, 202605_1_125_38, 202605_1_126_39, 202605_1_127_40, 202605_1_128_41, 202605_1_129_42, 202605_1_130_43, 202605_1_131_44, 202605_1_132_45, 202605_1_138_46, 202605_1_144_47, 202605_117_117_0, 202605_117_121_1, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0]
2026.05.06 08:48:34.913522 [ 32316 ] {} <Debug> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Removing 50 parts from filesystem (serially): Parts: [202605_1_116_33, 202605_1_121_34, 202605_1_122_35, 202605_1_123_36, 202605_1_124_37, 202605_1_125_38, 202605_1_126_39, 202605_1_127_40, 202605_1_128_41, 202605_1_129_42, 202605_1_130_43, 202605_1_131_44, 202605_1_132_45, 202605_1_138_46, 202605_1_144_47, 202605_117_117_0, 202605_117_121_1, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0]
2026.05.06 08:48:34.948942 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_116_33 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949024 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_121_34 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949057 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_122_35 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949091 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_123_36 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949132 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_124_37 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949166 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_125_38 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949197 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_126_39 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949230 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_127_40 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949262 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_128_41 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949294 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_129_42 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949326 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_130_43 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949358 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_131_44 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949391 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_132_45 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949423 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_138_46 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949454 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_1_144_47 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949497 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_117_117_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949528 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_117_121_1 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949559 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_118_118_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949590 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_119_119_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949622 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_120_120_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949653 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_121_121_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949685 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_122_122_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949716 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_123_123_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949748 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_124_124_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949784 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_125_125_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949815 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_126_126_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949847 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_127_127_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949879 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_128_128_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949911 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_129_129_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949943 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_130_130_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.949972 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_131_131_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950003 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_132_132_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950034 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_133_133_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950065 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_134_134_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950095 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_135_135_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950126 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_136_136_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950156 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_137_137_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950188 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_138_138_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950218 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_139_139_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950248 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_140_140_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950279 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_141_141_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950311 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_142_142_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950342 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_143_143_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950374 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_144_144_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950405 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_145_145_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950442 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_146_146_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950474 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_147_147_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950516 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_148_148_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950547 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_149_149_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950578 [ 32316 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): removePartsFinally: removing 202605_150_150_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.950729 [ 32316 ] {} <Debug> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Removing 50 parts from memory: Parts: [202605_1_116_33, 202605_1_121_34, 202605_1_122_35, 202605_1_123_36, 202605_1_124_37, 202605_1_125_38, 202605_1_126_39, 202605_1_127_40, 202605_1_128_41, 202605_1_129_42, 202605_1_130_43, 202605_1_131_44, 202605_1_132_45, 202605_1_138_46, 202605_1_144_47, 202605_117_117_0, 202605_117_121_1, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0]
2026.05.06 08:48:34.956163 [ 32311 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Found 54 old parts to remove. Parts: [202605_1_121_24, 202605_1_126_25, 202605_1_131_26, 202605_1_136_27, 202605_1_141_28, 202605_1_146_29, 202605_1_151_30, 202605_1_156_31, 202605_1_161_32, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0]
2026.05.06 08:48:34.956327 [ 32311 ] {} <Debug> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Removing 54 parts from filesystem (serially): Parts: [202605_1_121_24, 202605_1_126_25, 202605_1_131_26, 202605_1_136_27, 202605_1_141_28, 202605_1_146_29, 202605_1_151_30, 202605_1_156_31, 202605_1_161_32, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0]
2026.05.06 08:48:34.979274 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_1_121_24 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979333 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_1_126_25 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979360 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_1_131_26 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979387 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_1_136_27 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979413 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_1_141_28 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979441 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_1_146_29 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979468 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_1_151_30 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979502 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_1_156_31 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979528 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_1_161_32 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979555 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_122_122_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979581 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_123_123_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979632 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_124_124_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979659 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_125_125_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979686 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_126_126_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979716 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_127_127_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979741 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_128_128_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979766 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_129_129_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979792 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_130_130_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979817 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_131_131_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979842 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_132_132_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979875 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_133_133_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979902 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_134_134_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979927 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_135_135_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979952 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_136_136_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.979977 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_137_137_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980006 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_138_138_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980032 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_139_139_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980057 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_140_140_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980082 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_141_141_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980046 [ 32313 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Found 61 old parts to remove. Parts: [202605_1_126_58, 202605_1_127_59, 202605_1_128_60, 202605_1_129_61, 202605_1_130_62, 202605_1_131_63, 202605_1_132_64, 202605_1_138_65, 202605_1_144_66, 202605_1_150_67, 202605_1_155_68, 202605_1_156_69, 202605_1_157_70, 202605_1_163_71, 202605_1_167_72, 202605_1_168_73, 202605_1_169_74, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0, 202605_167_167_0, 202605_168_168_0, 202605_169_169_0, 202605_170_170_0]
2026.05.06 08:48:34.980107 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_142_142_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980161 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_143_143_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980190 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_144_144_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980214 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_145_145_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980240 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_146_146_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980265 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_147_147_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980295 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_148_148_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980320 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_149_149_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980345 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_150_150_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980371 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_151_151_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980396 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_152_152_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980421 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_153_153_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980446 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_154_154_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980470 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_155_155_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980509 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_156_156_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980535 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_157_157_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980559 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_158_158_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980532 [ 32313 ] {} <Debug> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Removing 61 parts from filesystem (serially): Parts: [202605_1_126_58, 202605_1_127_59, 202605_1_128_60, 202605_1_129_61, 202605_1_130_62, 202605_1_131_63, 202605_1_132_64, 202605_1_138_65, 202605_1_144_66, 202605_1_150_67, 202605_1_155_68, 202605_1_156_69, 202605_1_157_70, 202605_1_163_71, 202605_1_167_72, 202605_1_168_73, 202605_1_169_74, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0, 202605_167_167_0, 202605_168_168_0, 202605_169_169_0, 202605_170_170_0]
2026.05.06 08:48:34.980584 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_159_159_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980642 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_160_160_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980669 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_161_161_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980693 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_162_162_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980722 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_163_163_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980746 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_164_164_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980771 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_165_165_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980795 [ 32311 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): removePartsFinally: removing 202605_166_166_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:34.980913 [ 32311 ] {} <Debug> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Removing 54 parts from memory: Parts: [202605_1_121_24, 202605_1_126_25, 202605_1_131_26, 202605_1_136_27, 202605_1_141_28, 202605_1_146_29, 202605_1_151_30, 202605_1_156_31, 202605_1_161_32, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0]
2026.05.06 08:48:35.024645 [ 32314 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Found 36 old parts to remove. Parts: [202605_1_101_20, 202605_1_106_21, 202605_1_111_22, 202605_1_116_23, 202605_1_121_24, 202605_1_126_25, 202605_102_102_0, 202605_103_103_0, 202605_104_104_0, 202605_105_105_0, 202605_106_106_0, 202605_107_107_0, 202605_108_108_0, 202605_109_109_0, 202605_110_110_0, 202605_111_111_0, 202605_112_112_0, 202605_113_113_0, 202605_114_114_0, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0]
2026.05.06 08:48:35.025015 [ 32314 ] {} <Debug> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Removing 36 parts from filesystem (serially): Parts: [202605_1_101_20, 202605_1_106_21, 202605_1_111_22, 202605_1_116_23, 202605_1_121_24, 202605_1_126_25, 202605_102_102_0, 202605_103_103_0, 202605_104_104_0, 202605_105_105_0, 202605_106_106_0, 202605_107_107_0, 202605_108_108_0, 202605_109_109_0, 202605_110_110_0, 202605_111_111_0, 202605_112_112_0, 202605_113_113_0, 202605_114_114_0, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0]
2026.05.06 08:48:35.062430 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_126_58 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062506 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_127_59 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062543 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_128_60 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062581 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_129_61 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062610 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_130_62 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062644 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_131_63 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062673 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_132_64 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062706 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_138_65 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062745 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_144_66 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062784 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_150_67 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062818 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_155_68 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062853 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_156_69 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062888 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_157_70 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062922 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_163_71 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062951 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_167_72 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.062988 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_168_73 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063028 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_1_169_74 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063063 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_127_127_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063093 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_128_128_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063129 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_129_129_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063162 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_130_130_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063196 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_131_131_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063230 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_132_132_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063256 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_133_133_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063288 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_134_134_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063318 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_135_135_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063344 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_136_136_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063372 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_137_137_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063398 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_138_138_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063423 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_139_139_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063449 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_140_140_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063476 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_141_141_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063510 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_142_142_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063536 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_143_143_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063563 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_144_144_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.063590 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_145_145_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064360 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_146_146_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064397 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_147_147_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064424 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_148_148_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064451 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_149_149_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064477 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_150_150_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064513 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_151_151_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064539 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_152_152_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064566 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_153_153_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064593 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_154_154_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064619 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_155_155_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064650 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_156_156_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064677 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_157_157_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064702 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_158_158_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064729 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_159_159_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064755 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_160_160_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064781 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_161_161_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064807 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_162_162_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064833 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_163_163_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064859 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_164_164_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064884 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_165_165_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064910 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_166_166_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064936 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_167_167_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064962 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_168_168_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.064987 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_169_169_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.065013 [ 32313 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): removePartsFinally: removing 202605_170_170_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.065154 [ 32313 ] {} <Debug> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Removing 61 parts from memory: Parts: [202605_1_126_58, 202605_1_127_59, 202605_1_128_60, 202605_1_129_61, 202605_1_130_62, 202605_1_131_63, 202605_1_132_64, 202605_1_138_65, 202605_1_144_66, 202605_1_150_67, 202605_1_155_68, 202605_1_156_69, 202605_1_157_70, 202605_1_163_71, 202605_1_167_72, 202605_1_168_73, 202605_1_169_74, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0, 202605_132_132_0, 202605_133_133_0, 202605_134_134_0, 202605_135_135_0, 202605_136_136_0, 202605_137_137_0, 202605_138_138_0, 202605_139_139_0, 202605_140_140_0, 202605_141_141_0, 202605_142_142_0, 202605_143_143_0, 202605_144_144_0, 202605_145_145_0, 202605_146_146_0, 202605_147_147_0, 202605_148_148_0, 202605_149_149_0, 202605_150_150_0, 202605_151_151_0, 202605_152_152_0, 202605_153_153_0, 202605_154_154_0, 202605_155_155_0, 202605_156_156_0, 202605_157_157_0, 202605_158_158_0, 202605_159_159_0, 202605_160_160_0, 202605_161_161_0, 202605_162_162_0, 202605_163_163_0, 202605_164_164_0, 202605_165_165_0, 202605_166_166_0, 202605_167_167_0, 202605_168_168_0, 202605_169_169_0, 202605_170_170_0]
2026.05.06 08:48:35.067311 [ 31633 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal.
2026.05.06 08:48:35.067866 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_1_101_20 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.067949 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_1_106_21 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068002 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_1_111_22 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068060 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_1_116_23 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068111 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_1_121_24 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068164 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_1_126_25 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068220 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_102_102_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068272 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_103_103_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068324 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_104_104_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068381 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_105_105_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068432 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_106_106_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068498 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_107_107_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068556 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_108_108_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068606 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_109_109_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068654 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_110_110_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068702 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_111_111_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068759 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_112_112_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068809 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_113_113_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068864 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_114_114_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068914 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_115_115_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.068964 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_116_116_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069042 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_117_117_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069092 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_118_118_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069147 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_119_119_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069196 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_120_120_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069245 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_121_121_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069300 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_122_122_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069350 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_123_123_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069399 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_124_124_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069448 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_125_125_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069515 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_126_126_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069565 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_127_127_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069619 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_128_128_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069668 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_129_129_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069716 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_130_130_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069777 [ 32314 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): removePartsFinally: removing 202605_131_131_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.069920 [ 32314 ] {} <Debug> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Removing 36 parts from memory: Parts: [202605_1_101_20, 202605_1_106_21, 202605_1_111_22, 202605_1_116_23, 202605_1_121_24, 202605_1_126_25, 202605_102_102_0, 202605_103_103_0, 202605_104_104_0, 202605_105_105_0, 202605_106_106_0, 202605_107_107_0, 202605_108_108_0, 202605_109_109_0, 202605_110_110_0, 202605_111_111_0, 202605_112_112_0, 202605_113_113_0, 202605_114_114_0, 202605_115_115_0, 202605_116_116_0, 202605_117_117_0, 202605_118_118_0, 202605_119_119_0, 202605_120_120_0, 202605_121_121_0, 202605_122_122_0, 202605_123_123_0, 202605_124_124_0, 202605_125_125_0, 202605_126_126_0, 202605_127_127_0, 202605_128_128_0, 202605_129_129_0, 202605_130_130_0, 202605_131_131_0]
2026.05.06 08:48:35.072303 [ 31633 ] {} <Warning> Context: Server has databases (for example `test_01516`) with Ordinary engine, which was deprecated. To convert this database to the new Atomic engine, create a flag /var/lib/clickhouse/flags/convert_ordinary_to_atomic and make sure that ClickHouse has write permission for it. Example: sudo touch '/var/lib/clickhouse/flags/convert_ordinary_to_atomic' && sudo chmod 666 '/var/lib/clickhouse/flags/convert_ordinary_to_atomic'
2026.05.06 08:48:35.086851 [ 31633 ] {} <Information> DatabaseAtomic (01053_db): Metadata processed, database 01053_db has 1 tables and 1 dictionaries in total.
2026.05.06 08:48:35.088819 [ 31633 ] {} <Information> DatabaseAtomic (01778_db): Metadata processed, database 01778_db has 1 tables and 1 dictionaries in total.
2026.05.06 08:48:35.090071 [ 31633 ] {} <Information> DatabaseAtomic (01945.db): Metadata processed, database 01945.db has 1 tables and 1 dictionaries in total.
2026.05.06 08:48:35.090168 [ 31633 ] {} <Information> DatabaseAtomic (datasets): Metadata processed, database datasets has 0 tables and 0 dictionaries in total.
2026.05.06 08:48:35.090226 [ 31633 ] {} <Information> DatabaseAtomic (default): Metadata processed, database default has 0 tables and 0 dictionaries in total.
2026.05.06 08:48:35.090678 [ 31633 ] {} <Test> ZooKeeper: Availability zones from config: [''], client: ''
2026.05.06 08:48:35.091022 [ 31633 ] {} <Trace> ZooKeeperClient: Adding ZooKeeper host localhost:9181 ([::1]:9181), az: UNKNOWN, priority: 0
2026.05.06 08:48:35.091355 [ 32267 ] {} <Test> Poco: Queue size: 0, current threads: 0, threads in pool: 3, current connections: 0
2026.05.06 08:48:35.091686 [ 31640 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: [::1]:57482
2026.05.06 08:48:35.094986 [ 31640 ] {} <Information> KeeperTCPHandler: Requesting session ID for the new client
2026.05.06 08:48:35.095194 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:48:35.095435 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:35.095657 [ 32262 ] {} <Debug> KeeperStateMachine: Session ID response 5 with timeout 30000
2026.05.06 08:48:35.095833 [ 31640 ] {} <Information> KeeperTCPHandler: Received session ID 5
2026.05.06 08:48:35.095994 [ 31633 ] {} <Information> ZooKeeperClient: Connected to ZooKeeper at [::1]:9181 with session_id 5
2026.05.06 08:48:35.096825 [ 31633 ] {} <Information> ZooKeeperClient: Keeper feature flag FILTERED_LIST: disabled
2026.05.06 08:48:35.096892 [ 31633 ] {} <Information> ZooKeeperClient: Keeper feature flag MULTI_READ: disabled
2026.05.06 08:48:35.096920 [ 31633 ] {} <Information> ZooKeeperClient: Keeper feature flag CHECK_NOT_EXISTS: enabled
2026.05.06 08:48:35.096947 [ 31633 ] {} <Information> ZooKeeperClient: Keeper feature flag CREATE_IF_NOT_EXISTS: enabled
2026.05.06 08:48:35.096987 [ 31633 ] {} <Trace> ZooKeeper: Initialized, hosts: localhost:9181
2026.05.06 08:48:35.097284 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 75
2026.05.06 08:48:35.097549 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:35.098522 [ 31633 ] {} <Warning> DatabaseReplicated (rdb_test_13): Database rdb_test_13 exists locally, but its replica does not exist in ZooKeeper (/test/test_13/rdb/replicas/s1|r1). Assuming it was dropped, will not try to start it up
2026.05.06 08:48:35.099756 [ 31633 ] {} <Information> DatabaseReplicated (rdb_test_13): Metadata processed, database rdb_test_13 has 1 tables and 0 dictionaries in total.
2026.05.06 08:48:35.100983 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 66
2026.05.06 08:48:35.101194 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:35.101619 [ 31633 ] {} <Information> DatabaseReplicated (rdb_test_13_2): Metadata processed, database rdb_test_13_2 has 0 tables and 0 dictionaries in total.
2026.05.06 08:48:35.102976 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 66
2026.05.06 08:48:35.103322 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:35.103717 [ 31633 ] {} <Information> DatabaseReplicated (rdb_test_13_3): Metadata processed, database rdb_test_13_3 has 0 tables and 0 dictionaries in total.
2026.05.06 08:48:35.105116 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 66
2026.05.06 08:48:35.105454 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:35.105855 [ 31633 ] {} <Information> DatabaseReplicated (rdb_test_13_9): Metadata processed, database rdb_test_13_9 has 0 tables and 0 dictionaries in total.
2026.05.06 08:48:35.114958 [ 31633 ] {} <Information> DatabaseAtomic (shard_0): Metadata processed, database shard_0 has 4 tables and 0 dictionaries in total.
2026.05.06 08:48:35.124268 [ 31633 ] {} <Information> DatabaseAtomic (shard_1): Metadata processed, database shard_1 has 4 tables and 0 dictionaries in total.
2026.05.06 08:48:35.154202 [ 32315 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Found 11 old parts to remove. Parts: [202605_1_1_0, 202605_1_6_1, 202605_2_2_0, 202605_3_3_0, 202605_4_4_0, 202605_5_5_0, 202605_6_6_0, 202605_7_7_0, 202605_8_8_0, 202605_9_9_0, 202605_10_10_0]
2026.05.06 08:48:35.154376 [ 32315 ] {} <Debug> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Removing 11 parts from filesystem (serially): Parts: [202605_1_1_0, 202605_1_6_1, 202605_2_2_0, 202605_3_3_0, 202605_4_4_0, 202605_5_5_0, 202605_6_6_0, 202605_7_7_0, 202605_8_8_0, 202605_9_9_0, 202605_10_10_0]
2026.05.06 08:48:35.166427 [ 32315 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): removePartsFinally: removing 202605_1_1_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.166549 [ 32315 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): removePartsFinally: removing 202605_1_6_1 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.166615 [ 32315 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): removePartsFinally: removing 202605_2_2_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.166677 [ 32315 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): removePartsFinally: removing 202605_3_3_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.166739 [ 32315 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): removePartsFinally: removing 202605_4_4_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.166802 [ 32315 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): removePartsFinally: removing 202605_5_5_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.166863 [ 32315 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): removePartsFinally: removing 202605_6_6_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.166926 [ 32315 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): removePartsFinally: removing 202605_7_7_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.166987 [ 32315 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): removePartsFinally: removing 202605_8_8_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.167048 [ 32315 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): removePartsFinally: removing 202605_9_9_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.167109 [ 32315 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): removePartsFinally: removing 202605_10_10_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.167204 [ 32315 ] {} <Debug> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Removing 11 parts from memory: Parts: [202605_1_1_0, 202605_1_6_1, 202605_2_2_0, 202605_3_3_0, 202605_4_4_0, 202605_5_5_0, 202605_6_6_0, 202605_7_7_0, 202605_8_8_0, 202605_9_9_0, 202605_10_10_0]
2026.05.06 08:48:35.177067 [ 31633 ] {} <Information> DatabaseAtomic (test): Metadata processed, database test has 8 tables and 0 dictionaries in total.
2026.05.06 08:48:35.181369 [ 31633 ] {} <Information> DatabaseOrdinary (test_01516): Metadata processed, database test_01516 has 1 tables and 0 dictionaries in total.
2026.05.06 08:48:35.190508 [ 31633 ] {} <Information> DatabaseAtomic (test_02771): Metadata processed, database test_02771 has 1 tables and 0 dictionaries in total.
2026.05.06 08:48:35.263134 [ 31633 ] {} <Information> DatabaseAtomic (test_1): Metadata processed, database test_1 has 60 tables and 0 dictionaries in total.
2026.05.06 08:48:35.265451 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 59
2026.05.06 08:48:35.265781 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:35.266562 [ 31633 ] {} <Information> DatabaseReplicated (test_11): Metadata processed, database test_11 has 0 tables and 0 dictionaries in total.
2026.05.06 08:48:35.319478 [ 31633 ] {} <Information> DatabaseAtomic (test_13): Metadata processed, database test_13 has 54 tables and 5 dictionaries in total.
2026.05.06 08:48:35.339502 [ 31633 ] {} <Information> DatabaseAtomic (test_15): Metadata processed, database test_15 has 17 tables and 1 dictionaries in total.
2026.05.06 08:48:35.340300 [ 31633 ] {} <Information> DatabaseAtomic (test_2gxskyku_1): Metadata processed, database test_2gxskyku_1 has 1 tables and 0 dictionaries in total.
2026.05.06 08:48:35.349139 [ 31633 ] {} <Information> DatabaseAtomic (test_3): Metadata processed, database test_3 has 4 tables and 0 dictionaries in total.
2026.05.06 08:48:35.351035 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 57
2026.05.06 08:48:35.351557 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:35.352304 [ 31633 ] {} <Information> DatabaseReplicated (test_5): Metadata processed, database test_5 has 0 tables and 0 dictionaries in total.
2026.05.06 08:48:35.353239 [ 31633 ] {} <Information> DatabaseAtomic (test_jbjeaal4_1): Metadata processed, database test_jbjeaal4_1 has 1 tables and 0 dictionaries in total.
2026.05.06 08:48:35.353907 [ 31633 ] {} <Information> DatabaseAtomic (test_m91yt8r2_1): Metadata processed, database test_m91yt8r2_1 has 1 tables and 0 dictionaries in total.
2026.05.06 08:48:35.357194 [ 31633 ] {} <Information> DatabaseAtomic (test_mkf6rch8): Metadata processed, database test_mkf6rch8 has 1 tables and 0 dictionaries in total.
2026.05.06 08:48:35.363301 [ 31633 ] {} <Information> DatabaseAtomic (test_truncate_database): Metadata processed, database test_truncate_database has 11 tables and 1 dictionaries in total.
2026.05.06 08:48:35.373773 [ 31633 ] {} <Information> DatabaseAtomic (test_w50hoezt): Metadata processed, database test_w50hoezt has 1 tables and 0 dictionaries in total.
2026.05.06 08:48:35.386372 [ 31633 ] {} <Information> DatabaseAtomic (test_xfltqef0): Metadata processed, database test_xfltqef0 has 2 tables and 0 dictionaries in total.
2026.05.06 08:48:35.386623 [ 31633 ] {} <Information> TablesLoader: Parsed metadata of 185 tables in 27 databases in 0.310350428 sec
2026.05.06 08:48:35.436318 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.source_table_tiny_log has no dependencies (level 0)
2026.05.06 08:48:35.436440 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.source_table_memory has no dependencies (level 0)
2026.05.06 08:48:35.436508 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.source_table_dictionary has no dependencies (level 0)
2026.05.06 08:48:35.436549 [ 31633 ] {} <Trace> ReferentialDeps: Table test_15.mt_00763_1 has no dependencies (level 0)
2026.05.06 08:48:35.436593 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.simple_key_source_table_01862 has no dependencies (level 0)
2026.05.06 08:48:35.436653 [ 31633 ] {} <Trace> ReferentialDeps: Table test_15.`02183_dictionary_test_table` has no dependencies (level 0)
2026.05.06 08:48:35.436697 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.mt has no dependencies (level 0)
2026.05.06 08:48:35.436756 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.`02162_test_table` has no dependencies (level 0)
2026.05.06 08:48:35.436798 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.data_01509 has no dependencies (level 0)
2026.05.06 08:48:35.436833 [ 31633 ] {} <Trace> ReferentialDeps: Table test_1.a has no dependencies (level 0)
2026.05.06 08:48:35.436869 [ 31633 ] {} <Trace> ReferentialDeps: Table test.hits_dst has no dependencies (level 0)
2026.05.06 08:48:35.436927 [ 31633 ] {} <Trace> ReferentialDeps: Table test_1.b has no dependencies (level 0)
2026.05.06 08:48:35.436970 [ 31633 ] {} <Trace> ReferentialDeps: Table test.visits_null has no dependencies (level 0)
2026.05.06 08:48:35.437005 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.mv_source has no dependencies (level 0)
2026.05.06 08:48:35.437039 [ 31633 ] {} <Trace> ReferentialDeps: Table test_xfltqef0.mt has no dependencies (level 0)
2026.05.06 08:48:35.437076 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.source_table_log has no dependencies (level 0)
2026.05.06 08:48:35.437131 [ 31633 ] {} <Trace> ReferentialDeps: Table test_1.table_src has no dependencies (level 0)
2026.05.06 08:48:35.437174 [ 31633 ] {} <Trace> ReferentialDeps: Table test_1.table_dst has no dependencies (level 0)
2026.05.06 08:48:35.437216 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.source_table_merge_tree has no dependencies (level 0)
2026.05.06 08:48:35.437277 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.source_table_stripe_log has no dependencies (level 0)
2026.05.06 08:48:35.437320 [ 31633 ] {} <Trace> ReferentialDeps: Table test_1.`null` has no dependencies (level 0)
2026.05.06 08:48:35.437380 [ 31633 ] {} <Trace> ReferentialDeps: Table `01945.db`.test_dictionary_values has no dependencies (level 0)
2026.05.06 08:48:35.437416 [ 31633 ] {} <Trace> ReferentialDeps: Table test_1.tmp_01781 has no dependencies (level 0)
2026.05.06 08:48:35.437455 [ 31633 ] {} <Trace> ReferentialDeps: Table `01053_db`.table_for_dict has no dependencies (level 0)
2026.05.06 08:48:35.437499 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.mv_target has no dependencies (level 0)
2026.05.06 08:48:35.437536 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.simple_key_simple_attributes_source_table has no dependencies (level 0)
2026.05.06 08:48:35.437581 [ 31633 ] {} <Trace> ReferentialDeps: Table `01778_db`.hierarchy_source_table has no dependencies (level 0)
2026.05.06 08:48:35.437662 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.dest_view_tiny_log has 1 dependencies: test_truncate_database.source_table_tiny_log (level 1)
2026.05.06 08:48:35.437712 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.dest_view_memory has 1 dependencies: test_truncate_database.source_table_memory (level 1)
2026.05.06 08:48:35.437759 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.dest_dictionary has 1 dependencies: test_truncate_database.source_table_dictionary (level 1)
2026.05.06 08:48:35.437806 [ 31633 ] {} <Trace> ReferentialDeps: Table test_15.buffer_00763_1 has 1 dependencies: test_15.mt_00763_1 (level 1)
2026.05.06 08:48:35.437874 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.simple_key_hashed_dictionary_01862 has 1 dependencies: test_13.simple_key_source_table_01862 (level 1)
2026.05.06 08:48:35.437924 [ 31633 ] {} <Trace> ReferentialDeps: Table test_15.`02183_hashed_array_dictionary` has 1 dependencies: test_15.`02183_dictionary_test_table` (level 1)
2026.05.06 08:48:35.437988 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.buf has 1 dependencies: test_13.mt (level 1)
2026.05.06 08:48:35.438040 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.`02162_test_dictionary` has 1 dependencies: test_13.`02162_test_table` (level 1)
2026.05.06 08:48:35.438084 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.dict_01509 has 1 dependencies: test_13.data_01509 (level 1)
2026.05.06 08:48:35.438126 [ 31633 ] {} <Trace> ReferentialDeps: Table test.hits_buffer has 1 dependencies: test.hits_dst (level 1)
2026.05.06 08:48:35.438186 [ 31633 ] {} <Trace> ReferentialDeps: Table test_1.view_4 has 2 dependencies: test_1.a, test_1.b (level 1)
2026.05.06 08:48:35.438241 [ 31633 ] {} <Trace> ReferentialDeps: Table test.basic has 1 dependencies: test.visits_null (level 1)
2026.05.06 08:48:35.438301 [ 31633 ] {} <Trace> ReferentialDeps: Table test_xfltqef0.buf has 1 dependencies: test_xfltqef0.mt (level 1)
2026.05.06 08:48:35.438355 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.dest_view_log has 1 dependencies: test_truncate_database.source_table_log (level 1)
2026.05.06 08:48:35.438426 [ 31633 ] {} <Trace> ReferentialDeps: Table test_1.mv_b_even_even has 2 dependencies: test_1.table_dst, test_1.table_src (level 1)
2026.05.06 08:48:35.438474 [ 31633 ] {} <Trace> ReferentialDeps: Table test_1.mv_b_even has 2 dependencies: test_1.table_dst, test_1.table_src (level 1)
2026.05.06 08:48:35.438530 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.dest_view_merge_tree has 1 dependencies: test_truncate_database.source_table_merge_tree (level 1)
2026.05.06 08:48:35.438599 [ 31633 ] {} <Trace> ReferentialDeps: Table test_truncate_database.dest_view_stripe_log has 1 dependencies: test_truncate_database.source_table_stripe_log (level 1)
2026.05.06 08:48:35.438652 [ 31633 ] {} <Trace> ReferentialDeps: Table test_1.mv has 1 dependencies: test_1.`null` (level 1)
2026.05.06 08:48:35.438699 [ 31633 ] {} <Trace> ReferentialDeps: Table `01945.db`.test_dictionary has 1 dependencies: `01945.db`.test_dictionary_values (level 1)
2026.05.06 08:48:35.438746 [ 31633 ] {} <Trace> ReferentialDeps: Table test_1.dist_01781 has 1 dependencies: test_1.tmp_01781 (level 1)
2026.05.06 08:48:35.438791 [ 31633 ] {} <Trace> ReferentialDeps: Table `01053_db`.ssd_dict has 1 dependencies: `01053_db`.table_for_dict (level 1)
2026.05.06 08:48:35.438839 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.mv has 2 dependencies: test_13.mv_target, test_13.mv_source (level 1)
2026.05.06 08:48:35.438887 [ 31633 ] {} <Trace> ReferentialDeps: Table test_13.hashed_array_dictionary_simple_key_simple_attributes has 1 dependencies: test_13.simple_key_simple_attributes_source_table (level 1)
2026.05.06 08:48:35.438960 [ 31633 ] {} <Trace> ReferentialDeps: Table `01778_db`.hierarchy_flat_dictionary has 1 dependencies: `01778_db`.hierarchy_source_table (level 1)
2026.05.06 08:48:35.439145 [ 31633 ] {} <Trace> LoadingDeps: Table test_xfltqef0.buf has no dependencies (level 0)
2026.05.06 08:48:35.439251 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.source_table_memory has no dependencies (level 0)
2026.05.06 08:48:35.439300 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.dest_view_stripe_log has no dependencies (level 0)
2026.05.06 08:48:35.439338 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.dest_view_merge_tree has no dependencies (level 0)
2026.05.06 08:48:35.439376 [ 31633 ] {} <Trace> LoadingDeps: Table test_mkf6rch8.test_02481_mismatch_files has no dependencies (level 0)
2026.05.06 08:48:35.439412 [ 31633 ] {} <Trace> LoadingDeps: Table test_m91yt8r2_1.t has no dependencies (level 0)
2026.05.06 08:48:35.439449 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.source_table_dictionary has no dependencies (level 0)
2026.05.06 08:48:35.439494 [ 31633 ] {} <Trace> LoadingDeps: Table test_2gxskyku_1.t has no dependencies (level 0)
2026.05.06 08:48:35.439543 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.t_r_ok has no dependencies (level 0)
2026.05.06 08:48:35.439589 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.t1 has no dependencies (level 0)
2026.05.06 08:48:35.439654 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.t has no dependencies (level 0)
2026.05.06 08:48:35.439692 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.src has no dependencies (level 0)
2026.05.06 08:48:35.439728 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.t_proj_external has no dependencies (level 0)
2026.05.06 08:48:35.439763 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.partslost_1 has no dependencies (level 0)
2026.05.06 08:48:35.439798 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.merge_tree has no dependencies (level 0)
2026.05.06 08:48:35.439837 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.`02183_dictionary_test_table` has no dependencies (level 0)
2026.05.06 08:48:35.439872 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.ps has no dependencies (level 0)
2026.05.06 08:48:35.439907 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.test_table_comment has no dependencies (level 0)
2026.05.06 08:48:35.439965 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.dst has no dependencies (level 0)
2026.05.06 08:48:35.440008 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.test_table_2 has no dependencies (level 0)
2026.05.06 08:48:35.440043 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.dest_view_tiny_log has no dependencies (level 0)
2026.05.06 08:48:35.440078 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.test has no dependencies (level 0)
2026.05.06 08:48:35.440130 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.table_for_rename_replicated has no dependencies (level 0)
2026.05.06 08:48:35.440179 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.table_for_freeze_old_syntax has no dependencies (level 0)
2026.05.06 08:48:35.440219 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.table_for_freeze has no dependencies (level 0)
2026.05.06 08:48:35.440281 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.t_index_hypothesis has no dependencies (level 0)
2026.05.06 08:48:35.440322 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.t has no dependencies (level 0)
2026.05.06 08:48:35.440359 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.simple_key_source_table_01862 has no dependencies (level 0)
2026.05.06 08:48:35.440412 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.set_index_not__fuzz_0 has no dependencies (level 0)
2026.05.06 08:48:35.440457 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.rmt1 has no dependencies (level 0)
2026.05.06 08:48:35.440500 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.rename_table_multiple has no dependencies (level 0)
2026.05.06 08:48:35.440536 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.numbers_squashed has no dependencies (level 0)
2026.05.06 08:48:35.440571 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.mv_source has no dependencies (level 0)
2026.05.06 08:48:35.440605 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.mt has no dependencies (level 0)
2026.05.06 08:48:35.440639 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.my_events has no dependencies (level 0)
2026.05.06 08:48:35.440673 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.json_test has no dependencies (level 0)
2026.05.06 08:48:35.440708 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.dest_view_memory has no dependencies (level 0)
2026.05.06 08:48:35.440744 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.input_function_table_3 has no dependencies (level 0)
2026.05.06 08:48:35.440808 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.input_function_table_2 has no dependencies (level 0)
2026.05.06 08:48:35.440858 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.table_for_freeze_replicated has no dependencies (level 0)
2026.05.06 08:48:35.440893 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.storage has no dependencies (level 0)
2026.05.06 08:48:35.440927 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.h3_indexes has no dependencies (level 0)
2026.05.06 08:48:35.440961 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.x has no dependencies (level 0)
2026.05.06 08:48:35.440998 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.dict_02916_dictionary_access_test_13 has no dependencies (level 0)
2026.05.06 08:48:35.441033 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.defaulted has no dependencies (level 0)
2026.05.06 08:48:35.441069 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.source_table_merge_tree has no dependencies (level 0)
2026.05.06 08:48:35.441104 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.datetime_t has no dependencies (level 0)
2026.05.06 08:48:35.441138 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.date_t has no dependencies (level 0)
2026.05.06 08:48:35.441195 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.constrained has no dependencies (level 0)
2026.05.06 08:48:35.441230 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.collapsing_table has no dependencies (level 0)
2026.05.06 08:48:35.441265 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.dim_model has no dependencies (level 0)
2026.05.06 08:48:35.441305 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.alter_default has no dependencies (level 0)
2026.05.06 08:48:35.441339 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.A_M has no dependencies (level 0)
2026.05.06 08:48:35.441374 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.data_01670 has no dependencies (level 0)
2026.05.06 08:48:35.441423 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.input_function_table_2 has no dependencies (level 0)
2026.05.06 08:48:35.441476 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.bug_13492 has no dependencies (level 0)
2026.05.06 08:48:35.441548 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.A has no dependencies (level 0)
2026.05.06 08:48:35.441601 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.tab2 has no dependencies (level 0)
2026.05.06 08:48:35.441687 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.`02952_disjunction_optimization` has no dependencies (level 0)
2026.05.06 08:48:35.441746 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.`02162_test_table` has no dependencies (level 0)
2026.05.06 08:48:35.441795 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.mv_target has no dependencies (level 0)
2026.05.06 08:48:35.441833 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.table_with_column_ttl has no dependencies (level 0)
2026.05.06 08:48:35.441868 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.tmp_01781 has no dependencies (level 0)
2026.05.06 08:48:35.441903 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.test_without_merge has no dependencies (level 0)
2026.05.06 08:48:35.441938 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.dist_01781 has no dependencies (level 0)
2026.05.06 08:48:35.441972 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.test_s3_mt_dst has no dependencies (level 0)
2026.05.06 08:48:35.442007 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.test_s3_mt has no dependencies (level 0)
2026.05.06 08:48:35.442041 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.test_collation has no dependencies (level 0)
2026.05.06 08:48:35.442101 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.collapsing_suspicious_granularity has no dependencies (level 0)
2026.05.06 08:48:35.442155 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.t2 has no dependencies (level 0)
2026.05.06 08:48:35.442214 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.test2 has no dependencies (level 0)
2026.05.06 08:48:35.442254 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.table_when_b_even has no dependencies (level 0)
2026.05.06 08:48:35.442304 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.table_src has no dependencies (level 0)
2026.05.06 08:48:35.442359 [ 31633 ] {} <Trace> LoadingDeps: Table test_w50hoezt.test_constraints has no dependencies (level 0)
2026.05.06 08:48:35.442399 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.buffer_00763_1 has no dependencies (level 0)
2026.05.06 08:48:35.442434 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.tab1 has no dependencies (level 0)
2026.05.06 08:48:35.442473 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.t2 has no dependencies (level 0)
2026.05.06 08:48:35.442561 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.t1 has no dependencies (level 0)
2026.05.06 08:48:35.442615 [ 31633 ] {} <Trace> LoadingDeps: Table test_xfltqef0.mt has no dependencies (level 0)
2026.05.06 08:48:35.442651 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.t01565 has no dependencies (level 0)
2026.05.06 08:48:35.442685 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.t0 has no dependencies (level 0)
2026.05.06 08:48:35.442724 [ 31633 ] {} <Trace> LoadingDeps: Table test.`.inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8` has no dependencies (level 0)
2026.05.06 08:48:35.442761 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.t has no dependencies (level 0)
2026.05.06 08:48:35.442794 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.set has no dependencies (level 0)
2026.05.06 08:48:35.442828 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.xp has no dependencies (level 0)
2026.05.06 08:48:35.442862 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.seq has no dependencies (level 0)
2026.05.06 08:48:35.442902 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.select_final has no dependencies (level 0)
2026.05.06 08:48:35.442948 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.test_table has no dependencies (level 0)
2026.05.06 08:48:35.442994 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.mv has no dependencies (level 0)
2026.05.06 08:48:35.443028 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.mv_b_even_even has no dependencies (level 0)
2026.05.06 08:48:35.443063 [ 31633 ] {} <Trace> LoadingDeps: Table test_jbjeaal4_1.t has no dependencies (level 0)
2026.05.06 08:48:35.443097 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.mv has no dependencies (level 0)
2026.05.06 08:48:35.443133 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.parallel_replicas_plain has no dependencies (level 0)
2026.05.06 08:48:35.443169 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.insert_dedup_token2 has no dependencies (level 0)
2026.05.06 08:48:35.443225 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.decimal has no dependencies (level 0)
2026.05.06 08:48:35.443269 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.buf has no dependencies (level 0)
2026.05.06 08:48:35.443305 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.insert_dedup_token1 has no dependencies (level 0)
2026.05.06 08:48:35.443415 [ 31633 ] {} <Trace> LoadingDeps: Table test_3.distinct_in_order has no dependencies (level 0)
2026.05.06 08:48:35.443460 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.t1__fuzz_26 has no dependencies (level 0)
2026.05.06 08:48:35.443507 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.input_function_table_6 has no dependencies (level 0)
2026.05.06 08:48:35.443544 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.source_table_tiny_log has no dependencies (level 0)
2026.05.06 08:48:35.443580 [ 31633 ] {} <Trace> LoadingDeps: Table shard_0.from_1 has no dependencies (level 0)
2026.05.06 08:48:35.443627 [ 31633 ] {} <Trace> LoadingDeps: Table `01053_db`.table_for_dict has no dependencies (level 0)
2026.05.06 08:48:35.443665 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.dynamic_test_2 has no dependencies (level 0)
2026.05.06 08:48:35.443720 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.t_modify_to_nullable has no dependencies (level 0)
2026.05.06 08:48:35.443777 [ 31633 ] {} <Trace> LoadingDeps: Table test.basic has no dependencies (level 0)
2026.05.06 08:48:35.443823 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.dst has no dependencies (level 0)
2026.05.06 08:48:35.443885 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.input_function_table_5 has no dependencies (level 0)
2026.05.06 08:48:35.443922 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.table_gcd_codec_one_hundred_zeros has no dependencies (level 0)
2026.05.06 08:48:35.443959 [ 31633 ] {} <Trace> LoadingDeps: Table test_3.ordinary_distinct has no dependencies (level 0)
2026.05.06 08:48:35.443994 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.data2014 has no dependencies (level 0)
2026.05.06 08:48:35.444030 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.small has no dependencies (level 0)
2026.05.06 08:48:35.444064 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.rmt2 has no dependencies (level 0)
2026.05.06 08:48:35.444100 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.array_jaccard_index has no dependencies (level 0)
2026.05.06 08:48:35.444136 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.replicated_mutation_table has no dependencies (level 0)
2026.05.06 08:48:35.444172 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.dynamic_test_1 has no dependencies (level 0)
2026.05.06 08:48:35.444231 [ 31633 ] {} <Trace> LoadingDeps: Table test_3.parsing_with_names has no dependencies (level 0)
2026.05.06 08:48:35.444299 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.data2013 has no dependencies (level 0)
2026.05.06 08:48:35.444335 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.data has no dependencies (level 0)
2026.05.06 08:48:35.444385 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.data_01509 has no dependencies (level 0)
2026.05.06 08:48:35.444461 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.concurrent_alter_column has no dependencies (level 0)
2026.05.06 08:48:35.444521 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.agg_func_col has no dependencies (level 0)
2026.05.06 08:48:35.444556 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.agg has no dependencies (level 0)
2026.05.06 08:48:35.444601 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.temp has no dependencies (level 0)
2026.05.06 08:48:35.444657 [ 31633 ] {} <Trace> LoadingDeps: Table test_3.unit has no dependencies (level 0)
2026.05.06 08:48:35.444703 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.`.inner_id.8ca14fb2-e48e-4ec8-8fdf-854e1dc4cf8c` has no dependencies (level 0)
2026.05.06 08:48:35.444742 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.add_aggregate has no dependencies (level 0)
2026.05.06 08:48:35.444835 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.fact_cpc_clicks has no dependencies (level 0)
2026.05.06 08:48:35.444892 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.source_table_stripe_log has no dependencies (level 0)
2026.05.06 08:48:35.444937 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.`03811_sparse_column_aggregation_with_sum` has no dependencies (level 0)
2026.05.06 08:48:35.444979 [ 31633 ] {} <Trace> LoadingDeps: Table test_01516.primary_key_test has no dependencies (level 0)
2026.05.06 08:48:35.445029 [ 31633 ] {} <Trace> LoadingDeps: Table shard_0.data_02346 has no dependencies (level 0)
2026.05.06 08:48:35.445074 [ 31633 ] {} <Trace> LoadingDeps: Table test.visits has no dependencies (level 0)
2026.05.06 08:48:35.445122 [ 31633 ] {} <Trace> LoadingDeps: Table shard_1.from_1 has no dependencies (level 0)
2026.05.06 08:48:35.445170 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.s has no dependencies (level 0)
2026.05.06 08:48:35.445226 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.input_function_table_4 has no dependencies (level 0)
2026.05.06 08:48:35.445285 [ 31633 ] {} <Trace> LoadingDeps: Table shard_1.data_02346 has no dependencies (level 0)
2026.05.06 08:48:35.445336 [ 31633 ] {} <Trace> LoadingDeps: Table test.hits_s3 has no dependencies (level 0)
2026.05.06 08:48:35.445392 [ 31633 ] {} <Trace> LoadingDeps: Table test.avro has no dependencies (level 0)
2026.05.06 08:48:35.445438 [ 31633 ] {} <Trace> LoadingDeps: Table shard_1.to has no dependencies (level 0)
2026.05.06 08:48:35.445474 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.mydb has no dependencies (level 0)
2026.05.06 08:48:35.445527 [ 31633 ] {} <Trace> LoadingDeps: Table test.hits_dst has no dependencies (level 0)
2026.05.06 08:48:35.445576 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.tab has no dependencies (level 0)
2026.05.06 08:48:35.445649 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.table_dst has no dependencies (level 0)
2026.05.06 08:48:35.445706 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.simple_key_simple_attributes_source_table has no dependencies (level 0)
2026.05.06 08:48:35.445747 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.mv_b_even has no dependencies (level 0)
2026.05.06 08:48:35.445820 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.mt_with_pk has no dependencies (level 0)
2026.05.06 08:48:35.445874 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.json_columns has no dependencies (level 0)
2026.05.06 08:48:35.445909 [ 31633 ] {} <Trace> LoadingDeps: Table rdb_test_13.t3 has no dependencies (level 0)
2026.05.06 08:48:35.445944 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.`null` has no dependencies (level 0)
2026.05.06 08:48:35.445980 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.test_tuple_filter__fuzz_2 has no dependencies (level 0)
2026.05.06 08:48:35.446016 [ 31633 ] {} <Trace> LoadingDeps: Table test.hits_buffer has no dependencies (level 0)
2026.05.06 08:48:35.446050 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.src has no dependencies (level 0)
2026.05.06 08:48:35.446084 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.test1 has no dependencies (level 0)
2026.05.06 08:48:35.446119 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.mt_00763_1 has no dependencies (level 0)
2026.05.06 08:48:35.446154 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.override_test__fuzz_45 has no dependencies (level 0)
2026.05.06 08:48:35.446209 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.bug_67742 has no dependencies (level 0)
2026.05.06 08:48:35.446264 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.source_table_log has no dependencies (level 0)
2026.05.06 08:48:35.446302 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.B has no dependencies (level 0)
2026.05.06 08:48:35.446356 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.tab_00625 has no dependencies (level 0)
2026.05.06 08:48:35.446404 [ 31633 ] {} <Trace> LoadingDeps: Table shard_0.from_0 has no dependencies (level 0)
2026.05.06 08:48:35.446444 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.`.inner_id.9907701c-369a-410f-99dc-ca9afd271de1` has no dependencies (level 0)
2026.05.06 08:48:35.446489 [ 31633 ] {} <Trace> LoadingDeps: Table shard_1.from_0 has no dependencies (level 0)
2026.05.06 08:48:35.446523 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.view_4 has no dependencies (level 0)
2026.05.06 08:48:35.446557 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.test_table has no dependencies (level 0)
2026.05.06 08:48:35.446592 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.input_function_table_3 has no dependencies (level 0)
2026.05.06 08:48:35.446659 [ 31633 ] {} <Trace> LoadingDeps: Table `01778_db`.hierarchy_source_table has no dependencies (level 0)
2026.05.06 08:48:35.446711 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.test has no dependencies (level 0)
2026.05.06 08:48:35.446754 [ 31633 ] {} <Trace> LoadingDeps: Table test.hits has no dependencies (level 0)
2026.05.06 08:48:35.446792 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.dest_view_log has no dependencies (level 0)
2026.05.06 08:48:35.446830 [ 31633 ] {} <Trace> LoadingDeps: Table `01945.db`.test_dictionary_values has no dependencies (level 0)
2026.05.06 08:48:35.446866 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.data_01279 has no dependencies (level 0)
2026.05.06 08:48:35.446903 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.test_grouping_sets_predicate has no dependencies (level 0)
2026.05.06 08:48:35.446953 [ 31633 ] {} <Trace> LoadingDeps: Table test_1.t_missed_subcolumns has no dependencies (level 0)
2026.05.06 08:48:35.447004 [ 31633 ] {} <Trace> LoadingDeps: Table shard_0.to has no dependencies (level 0)
2026.05.06 08:48:35.447056 [ 31633 ] {} <Trace> LoadingDeps: Table test_02771.t has no dependencies (level 0)
2026.05.06 08:48:35.447193 [ 31633 ] {} <Trace> LoadingDeps: Table test_truncate_database.dest_dictionary has 1 dependencies: test_truncate_database.source_table_dictionary (level 1)
2026.05.06 08:48:35.447268 [ 31633 ] {} <Trace> LoadingDeps: Table test_15.`02183_hashed_array_dictionary` has 1 dependencies: test_15.`02183_dictionary_test_table` (level 1)
2026.05.06 08:48:35.447385 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.simple_key_hashed_dictionary_01862 has 1 dependencies: test_13.simple_key_source_table_01862 (level 1)
2026.05.06 08:48:35.447462 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.`02162_test_dictionary` has 1 dependencies: test_13.`02162_test_table` (level 1)
2026.05.06 08:48:35.447525 [ 31633 ] {} <Trace> LoadingDeps: Table `01053_db`.ssd_dict has 1 dependencies: `01053_db`.table_for_dict (level 1)
2026.05.06 08:48:35.447591 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.dict_01509 has 1 dependencies: test_13.data_01509 (level 1)
2026.05.06 08:48:35.447686 [ 31633 ] {} <Trace> LoadingDeps: Table test_13.hashed_array_dictionary_simple_key_simple_attributes has 1 dependencies: test_13.simple_key_simple_attributes_source_table (level 1)
2026.05.06 08:48:35.447760 [ 31633 ] {} <Trace> LoadingDeps: Table `01778_db`.hierarchy_flat_dictionary has 1 dependencies: `01778_db`.hierarchy_source_table (level 1)
2026.05.06 08:48:35.447825 [ 31633 ] {} <Trace> LoadingDeps: Table `01945.db`.test_dictionary has 1 dependencies: `01945.db`.test_dictionary_values (level 1)
2026.05.06 08:48:35.456837 [ 31633 ] {} <Information> loadMetadata: Start asynchronous loading of databases
2026.05.06 08:48:35.457615 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.dst' into BackgroundLoad
2026.05.06 08:48:35.457697 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.collapsing_suspicious_granularity' into BackgroundLoad
2026.05.06 08:48:35.457737 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.t01565' into BackgroundLoad
2026.05.06 08:48:35.457779 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.dict_01509' into BackgroundLoad
2026.05.06 08:48:35.457820 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.mv' into BackgroundLoad
2026.05.06 08:48:35.457849 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.t_r_ok' into BackgroundLoad
2026.05.06 08:48:35.457877 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.input_function_table_2' into BackgroundLoad
2026.05.06 08:48:35.457905 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_w50hoezt.test_constraints' into BackgroundLoad
2026.05.06 08:48:35.457945 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.datetime_t' into BackgroundLoad
2026.05.06 08:48:35.457989 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test..inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8' into BackgroundLoad
2026.05.06 08:48:35.458051 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.data2014' into BackgroundLoad
2026.05.06 08:48:35.458080 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.s' into BackgroundLoad
2026.05.06 08:48:35.458110 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_mkf6rch8.test_02481_mismatch_files' into BackgroundLoad
2026.05.06 08:48:35.458157 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.dest_view_stripe_log' into BackgroundLoad
2026.05.06 08:48:35.458200 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.test_tuple_filter__fuzz_2' into BackgroundLoad
2026.05.06 08:48:35.458242 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table shard_1.from_0' into BackgroundLoad
2026.05.06 08:48:35.458282 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.tmp_01781' into BackgroundLoad
2026.05.06 08:48:35.458317 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.test_without_merge' into BackgroundLoad
2026.05.06 08:48:35.458344 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.agg' into BackgroundLoad
2026.05.06 08:48:35.458386 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_xfltqef0.buf' into BackgroundLoad
2026.05.06 08:48:35.458423 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.source_table_tiny_log' into BackgroundLoad
2026.05.06 08:48:35.458449 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.decimal' into BackgroundLoad
2026.05.06 08:48:35.458475 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.src' into BackgroundLoad
2026.05.06 08:48:35.458518 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.constrained' into BackgroundLoad
2026.05.06 08:48:35.458560 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.t' into BackgroundLoad
2026.05.06 08:48:35.458599 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.data' into BackgroundLoad
2026.05.06 08:48:35.458628 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.t2' into BackgroundLoad
2026.05.06 08:48:35.458655 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.test_table' into BackgroundLoad
2026.05.06 08:48:35.458685 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.t_proj_external' into BackgroundLoad
2026.05.06 08:48:35.458744 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.dest_view_memory' into BackgroundLoad
2026.05.06 08:48:35.458799 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.table_dst' into BackgroundLoad
2026.05.06 08:48:35.458833 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table shard_0.from_0' into BackgroundLoad
2026.05.06 08:48:35.458860 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.source_table_memory' into BackgroundLoad
2026.05.06 08:48:35.458887 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_01516.primary_key_test' into BackgroundLoad
2026.05.06 08:48:35.458914 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.partslost_1' into BackgroundLoad
2026.05.06 08:48:35.458941 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.merge_tree' into BackgroundLoad
2026.05.06 08:48:35.458968 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.test_s3_mt_dst' into BackgroundLoad
2026.05.06 08:48:35.458996 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.table_gcd_codec_one_hundred_zeros' into BackgroundLoad
2026.05.06 08:48:35.459023 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.t_index_hypothesis' into BackgroundLoad
2026.05.06 08:48:35.459061 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.mv_b_even_even' into BackgroundLoad
2026.05.06 08:48:35.459097 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.test_table_comment' into BackgroundLoad
2026.05.06 08:48:35.459135 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1..inner_id.8ca14fb2-e48e-4ec8-8fdf-854e1dc4cf8c' into BackgroundLoad
2026.05.06 08:48:35.459178 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table shard_0.data_02346' into BackgroundLoad
2026.05.06 08:48:35.459220 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.input_function_table_3' into BackgroundLoad
2026.05.06 08:48:35.459258 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.test_s3_mt' into BackgroundLoad
2026.05.06 08:48:35.459291 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.test' into BackgroundLoad
2026.05.06 08:48:35.459318 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.mv' into BackgroundLoad
2026.05.06 08:48:35.459349 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.array_jaccard_index' into BackgroundLoad
2026.05.06 08:48:35.459389 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.dest_view_merge_tree' into BackgroundLoad
2026.05.06 08:48:35.459463 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.numbers_squashed' into BackgroundLoad
2026.05.06 08:48:35.459514 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.A' into BackgroundLoad
2026.05.06 08:48:35.459549 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.json_test' into BackgroundLoad
2026.05.06 08:48:35.459584 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.table_for_freeze' into BackgroundLoad
2026.05.06 08:48:35.459635 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.table_when_b_even' into BackgroundLoad
2026.05.06 08:48:35.459672 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.B' into BackgroundLoad
2026.05.06 08:48:35.459706 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.source_table_merge_tree' into BackgroundLoad
2026.05.06 08:48:35.459742 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.table_for_rename_replicated' into BackgroundLoad
2026.05.06 08:48:35.459778 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.test_table_2' into BackgroundLoad
2026.05.06 08:48:35.459812 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.test1' into BackgroundLoad
2026.05.06 08:48:35.459873 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.t0' into BackgroundLoad
2026.05.06 08:48:35.459906 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.ps' into BackgroundLoad
2026.05.06 08:48:35.459948 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_2gxskyku_1.t' into BackgroundLoad
2026.05.06 08:48:35.459990 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.test_collation' into BackgroundLoad
2026.05.06 08:48:35.460020 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.rename_table_multiple' into BackgroundLoad
2026.05.06 08:48:35.460046 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.table_for_freeze_old_syntax' into BackgroundLoad
2026.05.06 08:48:35.460072 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.mt' into BackgroundLoad
2026.05.06 08:48:35.460097 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.t1' into BackgroundLoad
2026.05.06 08:48:35.460123 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.rmt1' into BackgroundLoad
2026.05.06 08:48:35.460160 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.defaulted' into BackgroundLoad
2026.05.06 08:48:35.460227 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.my_events' into BackgroundLoad
2026.05.06 08:48:35.460257 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.dim_model' into BackgroundLoad
2026.05.06 08:48:35.460285 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.t1__fuzz_26' into BackgroundLoad
2026.05.06 08:48:35.460312 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.t2' into BackgroundLoad
2026.05.06 08:48:35.460338 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_m91yt8r2_1.t' into BackgroundLoad
2026.05.06 08:48:35.460366 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.table_for_freeze_replicated' into BackgroundLoad
2026.05.06 08:48:35.460393 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.t' into BackgroundLoad
2026.05.06 08:48:35.460419 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.mv_b_even' into BackgroundLoad
2026.05.06 08:48:35.460445 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.h3_indexes' into BackgroundLoad
2026.05.06 08:48:35.460491 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.data_01279' into BackgroundLoad
2026.05.06 08:48:35.460528 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table 01778_db.hierarchy_flat_dictionary' into BackgroundLoad
2026.05.06 08:48:35.460554 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.buf' into BackgroundLoad
2026.05.06 08:48:35.460579 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.mv_source' into BackgroundLoad
2026.05.06 08:48:35.460608 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.replicated_mutation_table' into BackgroundLoad
2026.05.06 08:48:35.460634 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.date_t' into BackgroundLoad
2026.05.06 08:48:35.460660 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.collapsing_table' into BackgroundLoad
2026.05.06 08:48:35.460687 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.data_01670' into BackgroundLoad
2026.05.06 08:48:35.460713 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.mydb' into BackgroundLoad
2026.05.06 08:48:35.460739 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.buffer_00763_1' into BackgroundLoad
2026.05.06 08:48:35.460778 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.dict_02916_dictionary_access_test_13' into BackgroundLoad
2026.05.06 08:48:35.460814 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.table_with_column_ttl' into BackgroundLoad
2026.05.06 08:48:35.460841 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.source_table_dictionary' into BackgroundLoad
2026.05.06 08:48:35.460875 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.x' into BackgroundLoad
2026.05.06 08:48:35.460901 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.input_function_table_2' into BackgroundLoad
2026.05.06 08:48:35.460929 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table 01778_db.hierarchy_source_table' into BackgroundLoad
2026.05.06 08:48:35.460956 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.t' into BackgroundLoad
2026.05.06 08:48:35.460982 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.dist_01781' into BackgroundLoad
2026.05.06 08:48:35.461009 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.set_index_not__fuzz_0' into BackgroundLoad
2026.05.06 08:48:35.461042 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.dest_dictionary' into BackgroundLoad
2026.05.06 08:48:35.461079 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.table_src' into BackgroundLoad
2026.05.06 08:48:35.461113 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.mt_00763_1' into BackgroundLoad
2026.05.06 08:48:35.461150 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.tab1' into BackgroundLoad
2026.05.06 08:48:35.461194 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test.avro' into BackgroundLoad
2026.05.06 08:48:35.461224 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.alter_default' into BackgroundLoad
2026.05.06 08:48:35.461251 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.t1' into BackgroundLoad
2026.05.06 08:48:35.461276 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.view_4' into BackgroundLoad
2026.05.06 08:48:35.461302 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_xfltqef0.mt' into BackgroundLoad
2026.05.06 08:48:35.461328 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.add_aggregate' into BackgroundLoad
2026.05.06 08:48:35.461354 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.xp' into BackgroundLoad
2026.05.06 08:48:35.461405 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table 01945.db.test_dictionary_values' into BackgroundLoad
2026.05.06 08:48:35.461433 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.mv_target' into BackgroundLoad
2026.05.06 08:48:35.461462 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.select_final' into BackgroundLoad
2026.05.06 08:48:35.461513 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_jbjeaal4_1.t' into BackgroundLoad
2026.05.06 08:48:35.461555 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.insert_dedup_token2' into BackgroundLoad
2026.05.06 08:48:35.461586 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.insert_dedup_token1' into BackgroundLoad
2026.05.06 08:48:35.461627 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.02952_disjunction_optimization' into BackgroundLoad
2026.05.06 08:48:35.461672 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.input_function_table_6' into BackgroundLoad
2026.05.06 08:48:35.461706 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.test_table' into BackgroundLoad
2026.05.06 08:48:35.461738 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.bug_13492' into BackgroundLoad
2026.05.06 08:48:35.461834 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.dynamic_test_2' into BackgroundLoad
2026.05.06 08:48:35.461880 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table shard_1.from_1' into BackgroundLoad
2026.05.06 08:48:35.461910 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.t_modify_to_nullable' into BackgroundLoad
2026.05.06 08:48:35.461938 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.json_columns' into BackgroundLoad
2026.05.06 08:48:35.461970 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.set' into BackgroundLoad
2026.05.06 08:48:35.461995 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table rdb_test_13.t3' into BackgroundLoad
2026.05.06 08:48:35.462023 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.parallel_replicas_plain' into BackgroundLoad
2026.05.06 08:48:35.462049 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.dst' into BackgroundLoad
2026.05.06 08:48:35.462075 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.tab_00625' into BackgroundLoad
2026.05.06 08:48:35.462102 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.input_function_table_5' into BackgroundLoad
2026.05.06 08:48:35.462166 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.input_function_table_4' into BackgroundLoad
2026.05.06 08:48:35.462206 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_3.ordinary_distinct' into BackgroundLoad
2026.05.06 08:48:35.462234 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.agg_func_col' into BackgroundLoad
2026.05.06 08:48:35.462261 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table shard_1.data_02346' into BackgroundLoad
2026.05.06 08:48:35.462287 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.bug_67742' into BackgroundLoad
2026.05.06 08:48:35.462313 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.data2013' into BackgroundLoad
2026.05.06 08:48:35.462338 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.small' into BackgroundLoad
2026.05.06 08:48:35.462364 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.mt_with_pk' into BackgroundLoad
2026.05.06 08:48:35.462391 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.rmt2' into BackgroundLoad
2026.05.06 08:48:35.462417 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.data_01509' into BackgroundLoad
2026.05.06 08:48:35.462469 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.dest_view_tiny_log' into BackgroundLoad
2026.05.06 08:48:35.462508 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table 01053_db.table_for_dict' into BackgroundLoad
2026.05.06 08:48:35.462535 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.concurrent_alter_column' into BackgroundLoad
2026.05.06 08:48:35.462572 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_3.unit' into BackgroundLoad
2026.05.06 08:48:35.462614 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.dynamic_test_1' into BackgroundLoad
2026.05.06 08:48:35.462654 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_3.distinct_in_order' into BackgroundLoad
2026.05.06 08:48:35.462683 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.fact_cpc_clicks' into BackgroundLoad
2026.05.06 08:48:35.462726 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_3.parsing_with_names' into BackgroundLoad
2026.05.06 08:48:35.462760 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.03811_sparse_column_aggregation_with_sum' into BackgroundLoad
2026.05.06 08:48:35.462804 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test.hits_dst' into BackgroundLoad
2026.05.06 08:48:35.462840 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table shard_0.to' into BackgroundLoad
2026.05.06 08:48:35.462865 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test.visits' into BackgroundLoad
2026.05.06 08:48:35.462903 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.seq' into BackgroundLoad
2026.05.06 08:48:35.462946 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.t_missed_subcolumns' into BackgroundLoad
2026.05.06 08:48:35.462982 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test.hits_s3' into BackgroundLoad
2026.05.06 08:48:35.463014 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.tab' into BackgroundLoad
2026.05.06 08:48:35.463039 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test.basic' into BackgroundLoad
2026.05.06 08:48:35.463065 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test.hits_buffer' into BackgroundLoad
2026.05.06 08:48:35.463091 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.src' into BackgroundLoad
2026.05.06 08:48:35.463132 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.override_test__fuzz_45' into BackgroundLoad
2026.05.06 08:48:35.463168 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.source_table_log' into BackgroundLoad
2026.05.06 08:48:35.463196 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table shard_0.from_1' into BackgroundLoad
2026.05.06 08:48:35.463222 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table 01053_db.ssd_dict' into BackgroundLoad
2026.05.06 08:48:35.463248 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table 01945.db.test_dictionary' into BackgroundLoad
2026.05.06 08:48:35.463275 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.test' into BackgroundLoad
2026.05.06 08:48:35.463300 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.null' into BackgroundLoad
2026.05.06 08:48:35.463329 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.input_function_table_3' into BackgroundLoad
2026.05.06 08:48:35.463376 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.simple_key_simple_attributes_source_table' into BackgroundLoad
2026.05.06 08:48:35.463419 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_02771.t' into BackgroundLoad
2026.05.06 08:48:35.463477 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.02162_test_dictionary' into BackgroundLoad
2026.05.06 08:48:35.463564 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test.hits' into BackgroundLoad
2026.05.06 08:48:35.463599 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.A_M' into BackgroundLoad
2026.05.06 08:48:35.463650 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.temp' into BackgroundLoad
2026.05.06 08:48:35.463689 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.test_grouping_sets_predicate' into BackgroundLoad
2026.05.06 08:48:35.463729 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.hashed_array_dictionary_simple_key_simple_attributes' into BackgroundLoad
2026.05.06 08:48:35.463774 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.02183_hashed_array_dictionary' into BackgroundLoad
2026.05.06 08:48:35.463817 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.storage' into BackgroundLoad
2026.05.06 08:48:35.463853 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.test2' into BackgroundLoad
2026.05.06 08:48:35.463883 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.simple_key_source_table_01862' into BackgroundLoad
2026.05.06 08:48:35.463952 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1.tab2' into BackgroundLoad
2026.05.06 08:48:35.463986 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.02162_test_table' into BackgroundLoad
2026.05.06 08:48:35.464016 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.source_table_stripe_log' into BackgroundLoad
2026.05.06 08:48:35.464044 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_15.02183_dictionary_test_table' into BackgroundLoad
2026.05.06 08:48:35.464072 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_1..inner_id.9907701c-369a-410f-99dc-ca9afd271de1' into BackgroundLoad
2026.05.06 08:48:35.464101 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_truncate_database.dest_view_log' into BackgroundLoad
2026.05.06 08:48:35.464135 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table shard_1.to' into BackgroundLoad
2026.05.06 08:48:35.464162 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'load table test_13.simple_key_hashed_dictionary_01862' into BackgroundLoad
2026.05.06 08:48:35.464218 [ 31633 ] {} <Debug> AsyncLoader: Change current priority: none -> 1
2026.05.06 08:48:35.464306 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #1 in BackgroundLoad
2026.05.06 08:48:35.464512 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #2 in BackgroundLoad
2026.05.06 08:48:35.464588 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #3 in BackgroundLoad
2026.05.06 08:48:35.464703 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #4 in BackgroundLoad
2026.05.06 08:48:35.464790 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #5 in BackgroundLoad
2026.05.06 08:48:35.464837 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #6 in BackgroundLoad
2026.05.06 08:48:35.464910 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #7 in BackgroundLoad
2026.05.06 08:48:35.464997 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #8 in BackgroundLoad
2026.05.06 08:48:35.465068 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #9 in BackgroundLoad
2026.05.06 08:48:35.465216 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #10 in BackgroundLoad
2026.05.06 08:48:35.465261 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #11 in BackgroundLoad
2026.05.06 08:48:35.465375 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #12 in BackgroundLoad
2026.05.06 08:48:35.465447 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #13 in BackgroundLoad
2026.05.06 08:48:35.465512 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #14 in BackgroundLoad
2026.05.06 08:48:35.465601 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #15 in BackgroundLoad
2026.05.06 08:48:35.465662 [ 31633 ] {} <Debug> AsyncLoader: Spawn loader worker #16 in BackgroundLoad
2026.05.06 08:48:35.466001 [ 32398 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.dst' in BackgroundLoad
2026.05.06 08:48:35.466090 [ 32398 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.dst
2026.05.06 08:48:35.466182 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.collapsing_suspicious_granularity' in BackgroundLoad
2026.05.06 08:48:35.466355 [ 32370 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.collapsing_suspicious_granularity
2026.05.06 08:48:35.466383 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.t01565' in BackgroundLoad
2026.05.06 08:48:35.466585 [ 32397 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.t01565
2026.05.06 08:48:35.466626 [ 32402 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.mv' in BackgroundLoad
2026.05.06 08:48:35.466754 [ 32402 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.mv
2026.05.06 08:48:35.466814 [ 32353 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.t_r_ok' in BackgroundLoad
2026.05.06 08:48:35.466942 [ 32353 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.t_r_ok
2026.05.06 08:48:35.467025 [ 32390 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.input_function_table_2' in BackgroundLoad
2026.05.06 08:48:35.467209 [ 32343 ] {} <Debug> AsyncLoader: Execute load job 'load table test_w50hoezt.test_constraints' in BackgroundLoad
2026.05.06 08:48:35.467220 [ 32390 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.input_function_table_2
2026.05.06 08:48:35.467331 [ 32343 ] {} <Trace> DatabaseAtomic (test_w50hoezt): Loading table test_w50hoezt.test_constraints
2026.05.06 08:48:35.467389 [ 32340 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.datetime_t' in BackgroundLoad
2026.05.06 08:48:35.467816 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.467820 [ 32340 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.datetime_t
2026.05.06 08:48:35.467861 [ 32403 ] {} <Debug> AsyncLoader: Execute load job 'load table test..inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8' in BackgroundLoad
2026.05.06 08:48:35.467893 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.468103 [ 32403 ] {} <Trace> DatabaseAtomic (test): Loading table test..inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8
2026.05.06 08:48:35.468107 [ 32397 ] {} <Test> DatabaseAtomic (test_1): Counting attached table t01565 to database test_1
2026.05.06 08:48:35.468162 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.data2014' in BackgroundLoad
2026.05.06 08:48:35.468418 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.t01565' with status OK
2026.05.06 08:48:35.468419 [ 32339 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.data2014
2026.05.06 08:48:35.468606 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.s' in BackgroundLoad
2026.05.06 08:48:35.468808 [ 32397 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.s
2026.05.06 08:48:35.468900 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_mkf6rch8.test_02481_mismatch_files' in BackgroundLoad
2026.05.06 08:48:35.469006 [ 32390 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.469085 [ 32395 ] {} <Trace> DatabaseAtomic (test_mkf6rch8): Loading table test_mkf6rch8.test_02481_mismatch_files
2026.05.06 08:48:35.469139 [ 32390 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.469163 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.dest_view_stripe_log' in BackgroundLoad
2026.05.06 08:48:35.469293 [ 32390 ] {} <Test> DatabaseAtomic (test_13): Counting attached table input_function_table_2 to database test_13
2026.05.06 08:48:35.469333 [ 32401 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.test_tuple_filter__fuzz_2' in BackgroundLoad
2026.05.06 08:48:35.469336 [ 32371 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.dest_view_stripe_log
2026.05.06 08:48:35.469425 [ 32401 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.test_tuple_filter__fuzz_2
2026.05.06 08:48:35.469507 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table shard_1.from_0' in BackgroundLoad
2026.05.06 08:48:35.469538 [ 32339 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.469837 [ 32374 ] {} <Trace> DatabaseAtomic (shard_1): Loading table shard_1.from_0
2026.05.06 08:48:35.469873 [ 32368 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.tmp_01781' in BackgroundLoad
2026.05.06 08:48:35.469982 [ 32368 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.tmp_01781
2026.05.06 08:48:35.470755 [ 32339 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.470904 [ 32339 ] {} <Test> DatabaseAtomic (test_1): Counting attached table data2014 to database test_1
2026.05.06 08:48:35.471021 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.471074 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.471130 [ 32397 ] {} <Test> DatabaseAtomic (test_1): Counting attached table s to database test_1
2026.05.06 08:48:35.471251 [ 32368 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.471292 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_01516.primary_key_test' into BackgrndStartup
2026.05.06 08:48:35.471348 [ 32368 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.471499 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database default' into BackgrndStartup
2026.05.06 08:48:35.471635 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database default' into BackgrndStartup
2026.05.06 08:48:35.471671 [ 32368 ] {} <Test> DatabaseAtomic (test_1): Counting attached table tmp_01781 to database test_1
2026.05.06 08:48:35.471681 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database rdb_test_13_3' into BackgrndStartup
2026.05.06 08:48:35.471764 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Replicated database rdb_test_13_3' into BackgrndStartup
2026.05.06 08:48:35.471811 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table shard_0.data_02346' into BackgrndStartup
2026.05.06 08:48:35.471860 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database shard_0' into BackgrndStartup
2026.05.06 08:48:35.471903 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database shard_0' into BackgrndStartup
2026.05.06 08:48:35.471936 [ 32402 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.471949 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_3' into BackgrndStartup
2026.05.06 08:48:35.472028 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_3.parsing_with_names' into BackgrndStartup
2026.05.06 08:48:35.472039 [ 32402 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.472142 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_3' into BackgrndStartup
2026.05.06 08:48:35.472188 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_m91yt8r2_1.t' into BackgrndStartup
2026.05.06 08:48:35.472217 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_m91yt8r2_1' into BackgrndStartup
2026.05.06 08:48:35.472225 [ 32402 ] {} <Test> DatabaseAtomic (test_1): Counting attached table mv to database test_1
2026.05.06 08:48:35.472261 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table shard_0.to' into BackgrndStartup
2026.05.06 08:48:35.472303 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_m91yt8r2_1' into BackgrndStartup
2026.05.06 08:48:35.472352 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.dest_view_memory' into BackgrndStartup
2026.05.06 08:48:35.472398 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.source_table_stripe_log' into BackgrndStartup
2026.05.06 08:48:35.472441 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.dest_view_tiny_log' into BackgrndStartup
2026.05.06 08:48:35.472479 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.source_table_log' into BackgrndStartup
2026.05.06 08:48:35.472610 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.source_table_memory' into BackgrndStartup
2026.05.06 08:48:35.472644 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_truncate_database' into BackgrndStartup
2026.05.06 08:48:35.472673 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.source_table_tiny_log' into BackgrndStartup
2026.05.06 08:48:35.472701 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_3.ordinary_distinct' into BackgrndStartup
2026.05.06 08:48:35.472729 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_3.distinct_in_order' into BackgrndStartup
2026.05.06 08:48:35.472756 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test.avro' into BackgrndStartup
2026.05.06 08:48:35.472785 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test.hits_s3' into BackgrndStartup
2026.05.06 08:48:35.472827 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test.visits' into BackgrndStartup
2026.05.06 08:48:35.472828 [ 32371 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.472866 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test.basic' into BackgrndStartup
2026.05.06 08:48:35.473067 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test..inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8' into BackgrndStartup
2026.05.06 08:48:35.473064 [ 32371 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.473099 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test' into BackgrndStartup
2026.05.06 08:48:35.473180 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test' into BackgrndStartup
2026.05.06 08:48:35.473210 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table 01053_db.ssd_dict' into BackgrndStartup
2026.05.06 08:48:35.473239 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database 01053_db' into BackgrndStartup
2026.05.06 08:48:35.473233 [ 32371 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table dest_view_stripe_log to database test_truncate_database
2026.05.06 08:48:35.473268 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table 01945.db.test_dictionary' into BackgrndStartup
2026.05.06 08:48:35.473317 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database 01945.db' into BackgrndStartup
2026.05.06 08:48:35.473354 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database 01945.db' into BackgrndStartup
2026.05.06 08:48:35.473419 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table shard_1.data_02346' into BackgrndStartup
2026.05.06 08:48:35.473507 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.source_table_dictionary' into BackgrndStartup
2026.05.06 08:48:35.473549 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.dest_view_stripe_log' into BackgrndStartup
2026.05.06 08:48:35.473578 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database rdb_test_13' into BackgrndStartup
2026.05.06 08:48:35.473606 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database rdb_test_13_9' into BackgrndStartup
2026.05.06 08:48:35.473633 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database rdb_test_13_9' into BackgrndStartup
2026.05.06 08:48:35.473660 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table shard_0.from_0' into BackgrndStartup
2026.05.06 08:48:35.473687 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.input_function_table_2' into BackgrndStartup
2026.05.06 08:48:35.473714 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.dest_view_log' into BackgrndStartup
2026.05.06 08:48:35.473741 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test.hits' into BackgrndStartup
2026.05.06 08:48:35.473802 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test.hits_buffer' into BackgrndStartup
2026.05.06 08:48:35.473861 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.input_function_table_4' into BackgrndStartup
2026.05.06 08:48:35.473888 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table 01778_db.hierarchy_source_table' into BackgrndStartup
2026.05.06 08:48:35.473915 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database 01778_db' into BackgrndStartup
2026.05.06 08:48:35.473942 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.dict_01509' into BackgrndStartup
2026.05.06 08:48:35.473971 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database 01778_db' into BackgrndStartup
2026.05.06 08:48:35.473998 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database datasets' into BackgrndStartup
2026.05.06 08:48:35.474032 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.json_test' into BackgrndStartup
2026.05.06 08:48:35.474059 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database datasets' into BackgrndStartup
2026.05.06 08:48:35.474086 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.date_t' into BackgrndStartup
2026.05.06 08:48:35.474233 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table rdb_test_13.t3' into BackgrndStartup
2026.05.06 08:48:35.474279 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Replicated database rdb_test_13' into BackgrndStartup
2026.05.06 08:48:35.474320 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_11' into BackgrndStartup
2026.05.06 08:48:35.474356 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table shard_1.from_0' into BackgrndStartup
2026.05.06 08:48:35.474393 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database rdb_test_13_2' into BackgrndStartup
2026.05.06 08:48:35.474435 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.datetime_t' into BackgrndStartup
2026.05.06 08:48:35.474475 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Replicated database rdb_test_13_2' into BackgrndStartup
2026.05.06 08:48:35.474523 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Replicated database rdb_test_13_9' into BackgrndStartup
2026.05.06 08:48:35.474556 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table shard_1.to' into BackgrndStartup
2026.05.06 08:48:35.474589 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database shard_1' into BackgrndStartup
2026.05.06 08:48:35.474702 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.ps' into BackgrndStartup
2026.05.06 08:48:35.474741 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.t_modify_to_nullable' into BackgrndStartup
2026.05.06 08:48:35.474776 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_02771.t' into BackgrndStartup
2026.05.06 08:48:35.474811 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_02771' into BackgrndStartup
2026.05.06 08:48:35.474847 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_2gxskyku_1' into BackgrndStartup
2026.05.06 08:48:35.474883 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.test_grouping_sets_predicate' into BackgrndStartup
2026.05.06 08:48:35.474919 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.test_table' into BackgrndStartup
2026.05.06 08:48:35.474953 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.mv_target' into BackgrndStartup
2026.05.06 08:48:35.474996 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.null' into BackgrndStartup
2026.05.06 08:48:35.475032 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.agg' into BackgrndStartup
2026.05.06 08:48:35.475133 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1..inner_id.8ca14fb2-e48e-4ec8-8fdf-854e1dc4cf8c' into BackgrndStartup
2026.05.06 08:48:35.475165 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.input_function_table_5' into BackgrndStartup
2026.05.06 08:48:35.475192 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.data2013' into BackgrndStartup
2026.05.06 08:48:35.475220 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.replicated_mutation_table' into BackgrndStartup
2026.05.06 08:48:35.475248 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.mv_b_even' into BackgrndStartup
2026.05.06 08:48:35.475282 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.table_gcd_codec_one_hundred_zeros' into BackgrndStartup
2026.05.06 08:48:35.475326 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.dim_model' into BackgrndStartup
2026.05.06 08:48:35.475378 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.test' into BackgrndStartup
2026.05.06 08:48:35.475408 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.x' into BackgrndStartup
2026.05.06 08:48:35.475473 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1..inner_id.9907701c-369a-410f-99dc-ca9afd271de1' into BackgrndStartup
2026.05.06 08:48:35.475546 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.tmp_01781' into BackgrndStartup
2026.05.06 08:48:35.475573 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.src' into BackgrndStartup
2026.05.06 08:48:35.475613 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.buf' into BackgrndStartup
2026.05.06 08:48:35.475640 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.t' into BackgrndStartup
2026.05.06 08:48:35.475666 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.decimal' into BackgrndStartup
2026.05.06 08:48:35.475695 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database rdb_test_13' into BackgrndStartup
2026.05.06 08:48:35.475723 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.select_final' into BackgrndStartup
2026.05.06 08:48:35.475749 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.data2014' into BackgrndStartup
2026.05.06 08:48:35.475777 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database 01053_db' into BackgrndStartup
2026.05.06 08:48:35.475838 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.fact_cpc_clicks' into BackgrndStartup
2026.05.06 08:48:35.475898 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.temp' into BackgrndStartup
2026.05.06 08:48:35.475926 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.bug_67742' into BackgrndStartup
2026.05.06 08:48:35.475953 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.t2' into BackgrndStartup
2026.05.06 08:48:35.475980 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.insert_dedup_token2' into BackgrndStartup
2026.05.06 08:48:35.476007 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_jbjeaal4_1.t' into BackgrndStartup
2026.05.06 08:48:35.476033 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.table_src' into BackgrndStartup
2026.05.06 08:48:35.476060 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.t1' into BackgrndStartup
2026.05.06 08:48:35.476089 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.t0' into BackgrndStartup
2026.05.06 08:48:35.476117 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.test_table_2' into BackgrndStartup
2026.05.06 08:48:35.476176 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.table_when_b_even' into BackgrndStartup
2026.05.06 08:48:35.476234 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.s' into BackgrndStartup
2026.05.06 08:48:35.476264 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database rdb_test_13_3' into BackgrndStartup
2026.05.06 08:48:35.476310 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.table_with_column_ttl' into BackgrndStartup
2026.05.06 08:48:35.476352 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.set' into BackgrndStartup
2026.05.06 08:48:35.476382 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.input_function_table_3' into BackgrndStartup
2026.05.06 08:48:35.476409 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_3.unit' into BackgrndStartup
2026.05.06 08:48:35.476437 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.input_function_table_6' into BackgrndStartup
2026.05.06 08:48:35.476470 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.dist_01781' into BackgrndStartup
2026.05.06 08:48:35.476508 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.concurrent_alter_column' into BackgrndStartup
2026.05.06 08:48:35.476609 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.test_s3_mt' into BackgrndStartup
2026.05.06 08:48:35.476635 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.tab2' into BackgrndStartup
2026.05.06 08:48:35.476663 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.02952_disjunction_optimization' into BackgrndStartup
2026.05.06 08:48:35.476690 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.mv_b_even_even' into BackgrndStartup
2026.05.06 08:48:35.476717 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.table_dst' into BackgrndStartup
2026.05.06 08:48:35.476744 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_1' into BackgrndStartup
2026.05.06 08:48:35.476771 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_1' into BackgrndStartup
2026.05.06 08:48:35.476798 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_11' into BackgrndStartup
2026.05.06 08:48:35.476825 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test.hits_dst' into BackgrndStartup
2026.05.06 08:48:35.476856 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Replicated database test_11' into BackgrndStartup
2026.05.06 08:48:35.476950 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.override_test__fuzz_45' into BackgrndStartup
2026.05.06 08:48:35.476976 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.test_without_merge' into BackgrndStartup
2026.05.06 08:48:35.477007 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.tab_00625' into BackgrndStartup
2026.05.06 08:48:35.477035 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.test_tuple_filter__fuzz_2' into BackgrndStartup
2026.05.06 08:48:35.477061 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.defaulted' into BackgrndStartup
2026.05.06 08:48:35.477089 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.03811_sparse_column_aggregation_with_sum' into BackgrndStartup
2026.05.06 08:48:35.477118 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_mkf6rch8' into BackgrndStartup
2026.05.06 08:48:35.477146 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.data_01509' into BackgrndStartup
2026.05.06 08:48:35.477173 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.table_for_freeze' into BackgrndStartup
2026.05.06 08:48:35.477204 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.t2' into BackgrndStartup
2026.05.06 08:48:35.477285 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.A' into BackgrndStartup
2026.05.06 08:48:35.477330 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.view_4' into BackgrndStartup
2026.05.06 08:48:35.477372 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.numbers_squashed' into BackgrndStartup
2026.05.06 08:48:35.477401 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database rdb_test_13_2' into BackgrndStartup
2026.05.06 08:48:35.477428 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.input_function_table_2' into BackgrndStartup
2026.05.06 08:48:35.477456 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.hashed_array_dictionary_simple_key_simple_attributes' into BackgrndStartup
2026.05.06 08:48:35.477493 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.add_aggregate' into BackgrndStartup
2026.05.06 08:48:35.477520 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_2gxskyku_1.t' into BackgrndStartup
2026.05.06 08:48:35.477553 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.mt_with_pk' into BackgrndStartup
2026.05.06 08:48:35.477581 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.simple_key_hashed_dictionary_01862' into BackgrndStartup
2026.05.06 08:48:35.477659 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.02162_test_table' into BackgrndStartup
2026.05.06 08:48:35.477687 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_xfltqef0.mt' into BackgrndStartup
2026.05.06 08:48:35.477715 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.set_index_not__fuzz_0' into BackgrndStartup
2026.05.06 08:48:35.477742 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_13' into BackgrndStartup
2026.05.06 08:48:35.477769 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.dest_view_merge_tree' into BackgrndStartup
2026.05.06 08:48:35.477796 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.bug_13492' into BackgrndStartup
2026.05.06 08:48:35.477823 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.mv' into BackgrndStartup
2026.05.06 08:48:35.477849 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.xp' into BackgrndStartup
2026.05.06 08:48:35.477877 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.table_for_freeze_replicated' into BackgrndStartup
2026.05.06 08:48:35.477931 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.dynamic_test_2' into BackgrndStartup
2026.05.06 08:48:35.477979 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.json_columns' into BackgrndStartup
2026.05.06 08:48:35.478007 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.dest_dictionary' into BackgrndStartup
2026.05.06 08:48:35.478034 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.test1' into BackgrndStartup
2026.05.06 08:48:35.478060 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_01516' into BackgrndStartup
2026.05.06 08:48:35.478088 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.collapsing_table' into BackgrndStartup
2026.05.06 08:48:35.478117 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.input_function_table_3' into BackgrndStartup
2026.05.06 08:48:35.478143 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.seq' into BackgrndStartup
2026.05.06 08:48:35.478173 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_2gxskyku_1' into BackgrndStartup
2026.05.06 08:48:35.478201 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.dict_02916_dictionary_access_test_13' into BackgrndStartup
2026.05.06 08:48:35.478268 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.t_proj_external' into BackgrndStartup
2026.05.06 08:48:35.478343 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.agg_func_col' into BackgrndStartup
2026.05.06 08:48:35.478375 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.alter_default' into BackgrndStartup
2026.05.06 08:48:35.478402 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.mydb' into BackgrndStartup
2026.05.06 08:48:35.478429 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.rename_table_multiple' into BackgrndStartup
2026.05.06 08:48:35.478457 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.data_01670' into BackgrndStartup
2026.05.06 08:48:35.478493 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.src' into BackgrndStartup
2026.05.06 08:48:35.478520 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.my_events' into BackgrndStartup
2026.05.06 08:48:35.478552 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.t1' into BackgrndStartup
2026.05.06 08:48:35.478579 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.table_for_freeze_old_syntax' into BackgrndStartup
2026.05.06 08:48:35.478638 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.collapsing_suspicious_granularity' into BackgrndStartup
2026.05.06 08:48:35.478734 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_truncate_database.source_table_merge_tree' into BackgrndStartup
2026.05.06 08:48:35.478763 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.storage' into BackgrndStartup
2026.05.06 08:48:35.478791 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_jbjeaal4_1' into BackgrndStartup
2026.05.06 08:48:35.478824 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.rmt1' into BackgrndStartup
2026.05.06 08:48:35.478851 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.t_missed_subcolumns' into BackgrndStartup
2026.05.06 08:48:35.478878 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table 01053_db.table_for_dict' into BackgrndStartup
2026.05.06 08:48:35.478907 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.table_for_rename_replicated' into BackgrndStartup
2026.05.06 08:48:35.478934 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.data' into BackgrndStartup
2026.05.06 08:48:35.478973 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.t01565' into BackgrndStartup
2026.05.06 08:48:35.479069 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Replicated database test_5' into BackgrndStartup
2026.05.06 08:48:35.479097 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.mt_00763_1' into BackgrndStartup
2026.05.06 08:48:35.479127 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.buffer_00763_1' into BackgrndStartup
2026.05.06 08:48:35.479156 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.tab1' into BackgrndStartup
2026.05.06 08:48:35.479184 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.02183_hashed_array_dictionary' into BackgrndStartup
2026.05.06 08:48:35.479211 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.data_01279' into BackgrndStartup
2026.05.06 08:48:35.479238 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_xfltqef0' into BackgrndStartup
2026.05.06 08:48:35.479269 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table shard_0.from_1' into BackgrndStartup
2026.05.06 08:48:35.479297 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.dst' into BackgrndStartup
2026.05.06 08:48:35.479337 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.test_s3_mt_dst' into BackgrndStartup
2026.05.06 08:48:35.479460 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table 01945.db.test_dictionary_values' into BackgrndStartup
2026.05.06 08:48:35.479505 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.A_M' into BackgrndStartup
2026.05.06 08:48:35.479533 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.test_table_comment' into BackgrndStartup
2026.05.06 08:48:35.479559 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.partslost_1' into BackgrndStartup
2026.05.06 08:48:35.479586 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.t' into BackgrndStartup
2026.05.06 08:48:35.479627 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.simple_key_source_table_01862' into BackgrndStartup
2026.05.06 08:48:35.479654 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.B' into BackgrndStartup
2026.05.06 08:48:35.479685 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.t' into BackgrndStartup
2026.05.06 08:48:35.479711 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.tab' into BackgrndStartup
2026.05.06 08:48:35.479739 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.parallel_replicas_plain' into BackgrndStartup
2026.05.06 08:48:35.479831 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_w50hoezt' into BackgrndStartup
2026.05.06 08:48:35.479858 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.mt' into BackgrndStartup
2026.05.06 08:48:35.479885 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_13' into BackgrndStartup
2026.05.06 08:48:35.479913 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_mkf6rch8.test_02481_mismatch_files' into BackgrndStartup
2026.05.06 08:48:35.479941 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.02183_dictionary_test_table' into BackgrndStartup
2026.05.06 08:48:35.479968 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.t1__fuzz_26' into BackgrndStartup
2026.05.06 08:48:35.479997 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_truncate_database' into BackgrndStartup
2026.05.06 08:48:35.480023 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.merge_tree' into BackgrndStartup
2026.05.06 08:48:35.480049 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.rmt2' into BackgrndStartup
2026.05.06 08:48:35.480131 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database shard_1' into BackgrndStartup
2026.05.06 08:48:35.480186 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.test' into BackgrndStartup
2026.05.06 08:48:35.480212 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_5' into BackgrndStartup
2026.05.06 08:48:35.480239 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_15' into BackgrndStartup
2026.05.06 08:48:35.480271 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_mkf6rch8' into BackgrndStartup
2026.05.06 08:48:35.480318 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.mv' into BackgrndStartup
2026.05.06 08:48:35.480354 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_w50hoezt' into BackgrndStartup
2026.05.06 08:48:35.480383 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.test_collation' into BackgrndStartup
2026.05.06 08:48:35.480410 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.test2' into BackgrndStartup
2026.05.06 08:48:35.480438 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.simple_key_simple_attributes_source_table' into BackgrndStartup
2026.05.06 08:48:35.480514 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_15' into BackgrndStartup
2026.05.06 08:48:35.480568 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.insert_dedup_token1' into BackgrndStartup
2026.05.06 08:48:35.480595 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.dst' into BackgrndStartup
2026.05.06 08:48:35.480622 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_w50hoezt.test_constraints' into BackgrndStartup
2026.05.06 08:48:35.480648 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Atomic database test_5' into BackgrndStartup
2026.05.06 08:48:35.480675 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.small' into BackgrndStartup
2026.05.06 08:48:35.480703 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table 01778_db.hierarchy_flat_dictionary' into BackgrndStartup
2026.05.06 08:48:35.480731 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.h3_indexes' into BackgrndStartup
2026.05.06 08:48:35.480764 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_02771' into BackgrndStartup
2026.05.06 08:48:35.480793 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.array_jaccard_index' into BackgrndStartup
2026.05.06 08:48:35.480852 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.02162_test_dictionary' into BackgrndStartup
2026.05.06 08:48:35.480907 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.t_index_hypothesis' into BackgrndStartup
2026.05.06 08:48:35.480934 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.t_r_ok' into BackgrndStartup
2026.05.06 08:48:35.480961 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table shard_1.from_1' into BackgrndStartup
2026.05.06 08:48:35.480991 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_15.test_table' into BackgrndStartup
2026.05.06 08:48:35.481019 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.mv_source' into BackgrndStartup
2026.05.06 08:48:35.481047 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_jbjeaal4_1' into BackgrndStartup
2026.05.06 08:48:35.481073 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_xfltqef0.buf' into BackgrndStartup
2026.05.06 08:48:35.481100 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_13.constrained' into BackgrndStartup
2026.05.06 08:48:35.481126 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup table test_1.dynamic_test_1' into BackgrndStartup
2026.05.06 08:48:35.481218 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup Ordinary database test_xfltqef0' into BackgrndStartup
2026.05.06 08:48:35.481648 [ 32370 ] {} <Debug> test_13.collapsing_suspicious_granularity (3f575314-61b2-4c2a-87df-ac628b3fa131): Loading data parts
2026.05.06 08:48:35.481779 [ 32340 ] {} <Debug> test_13.datetime_t (3bce9c68-e7c8-47d6-b419-21a9540b5133): Loading data parts
2026.05.06 08:48:35.481854 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.test_without_merge' in BackgroundLoad
2026.05.06 08:48:35.482033 [ 32381 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.test_without_merge
2026.05.06 08:48:35.482040 [ 32398 ] {} <Debug> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Loading data parts
2026.05.06 08:48:35.482036 [ 32395 ] {} <Information> AWSClient: ClientConfiguration: User agent is overridden in the config: ClickHouse 24.8.14.10546.altinitytest (altinity build)
2026.05.06 08:48:35.482063 [ 32390 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.input_function_table_2' with status OK
2026.05.06 08:48:35.482271 [ 32370 ] {} <Test> test_13.collapsing_suspicious_granularity (3f575314-61b2-4c2a-87df-ac628b3fa131): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.482356 [ 32390 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.agg' in BackgroundLoad
2026.05.06 08:48:35.482407 [ 32370 ] {} <Trace> test_13.collapsing_suspicious_granularity (3f575314-61b2-4c2a-87df-ac628b3fa131): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.482472 [ 32390 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.agg
2026.05.06 08:48:35.482476 [ 32340 ] {} <Test> test_13.datetime_t (3bce9c68-e7c8-47d6-b419-21a9540b5133): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.482541 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.data2014' with status OK
2026.05.06 08:48:35.482632 [ 32340 ] {} <Debug> test_13.datetime_t (3bce9c68-e7c8-47d6-b419-21a9540b5133): There are no data parts
2026.05.06 08:48:35.482870 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_xfltqef0.buf' in BackgroundLoad
2026.05.06 08:48:35.482903 [ 32378 ] {} <Trace> test_13.collapsing_suspicious_granularity (3f575314-61b2-4c2a-87df-ac628b3fa131): Loading Active part all_1_1_1 from disk default
2026.05.06 08:48:35.483005 [ 32339 ] {} <Trace> DatabaseAtomic (test_xfltqef0): Loading table test_xfltqef0.buf
2026.05.06 08:48:35.483019 [ 32398 ] {} <Test> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.483048 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.s' with status OK
2026.05.06 08:48:35.483246 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.source_table_tiny_log' in BackgroundLoad
2026.05.06 08:48:35.483256 [ 32398 ] {} <Debug> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): There are no data parts
2026.05.06 08:48:35.483363 [ 32397 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.source_table_tiny_log
2026.05.06 08:48:35.483397 [ 32368 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.tmp_01781' with status OK
2026.05.06 08:48:35.483712 [ 32368 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.decimal' in BackgroundLoad
2026.05.06 08:48:35.483700 [ 32398 ] {} <Information> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.483716 [ 32340 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.483823 [ 32368 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.decimal
2026.05.06 08:48:35.483879 [ 32340 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.483987 [ 32340 ] {} <Test> DatabaseAtomic (test_13): Counting attached table datetime_t to database test_13
2026.05.06 08:48:35.484583 [ 32402 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.mv' with status OK
2026.05.06 08:48:35.484684 [ 32368 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.484720 [ 32402 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.src' in BackgroundLoad
2026.05.06 08:48:35.484788 [ 32368 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.484821 [ 32402 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.src
2026.05.06 08:48:35.485036 [ 32398 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.485074 [ 32368 ] {} <Test> DatabaseAtomic (test_1): Counting attached table decimal to database test_1
2026.05.06 08:48:35.485071 [ 31633 ] {} <Information> UserDefinedSQLObjectsLoaderFromDisk: Loading user defined objects from /var/lib/clickhouse/user_defined/
2026.05.06 08:48:35.485122 [ 32398 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.485206 [ 32339 ] {} <Debug> DatabaseAtomic (test_xfltqef0): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.485198 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk azure
2026.05.06 08:48:35.485272 [ 32398 ] {} <Test> DatabaseAtomic (test_15): Counting attached table dst to database test_15
2026.05.06 08:48:35.485315 [ 32339 ] {} <Debug> DatabaseAtomic (test_xfltqef0): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.485330 [ 31633 ] {} <Debug> UserDefinedSQLObjectsLoaderFromDisk: The directory for user defined objects (/var/lib/clickhouse/user_defined/) does not exist: nothing to load
2026.05.06 08:48:35.485366 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk backups
2026.05.06 08:48:35.485404 [ 32368 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.decimal' with status OK
2026.05.06 08:48:35.485415 [ 32339 ] {} <Test> DatabaseAtomic (test_xfltqef0): Counting attached table buf to database test_xfltqef0
2026.05.06 08:48:35.485467 [ 31633 ] {} <Debug> Application: Loaded metadata.
2026.05.06 08:48:35.485508 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk cached_azure
2026.05.06 08:48:35.485541 [ 32368 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.constrained' in BackgroundLoad
2026.05.06 08:48:35.485628 [ 31633 ] {} <Information> Application: Tasks stats provider: procfs
2026.05.06 08:48:35.485752 [ 32368 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.constrained
2026.05.06 08:48:35.485908 [ 32397 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.485957 [ 32340 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.datetime_t' with status OK
2026.05.06 08:48:35.486007 [ 32397 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.486076 [ 32340 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.t' in BackgroundLoad
2026.05.06 08:48:35.486105 [ 32397 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table source_table_tiny_log to database test_truncate_database
2026.05.06 08:48:35.486173 [ 32340 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.t
2026.05.06 08:48:35.486282 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.source_table_tiny_log' with status OK
2026.05.06 08:48:35.486366 [ 31633 ] {} <Debug> DatabaseAtomic (system): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.486400 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.data' in BackgroundLoad
2026.05.06 08:48:35.486573 [ 31633 ] {} <Debug> DatabaseAtomic (system): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.486677 [ 32397 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.data
2026.05.06 08:48:35.486781 [ 32401 ] {} <Debug> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loading data parts
2026.05.06 08:48:35.486879 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_xfltqef0.buf' with status OK
2026.05.06 08:48:35.487002 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.t2' in BackgroundLoad
2026.05.06 08:48:35.487090 [ 32339 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.t2
2026.05.06 08:48:35.487786 [ 32401 ] {} <Test> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.487848 [ 32401 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Will load 10 parts using up to 64 threads
2026.05.06 08:48:35.488096 [ 32365 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loading Active part 19728_6_6_0 from disk default
2026.05.06 08:48:35.488148 [ 32376 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loading Active part 19731_9_9_0 from disk default
2026.05.06 08:48:35.488191 [ 32387 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loading Active part 19724_2_2_0 from disk default
2026.05.06 08:48:35.488441 [ 32374 ] {} <Debug> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Loading data parts
2026.05.06 08:48:35.488668 [ 32344 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loading Active part 19730_8_8_0 from disk default
2026.05.06 08:48:35.488727 [ 32374 ] {} <Test> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.488768 [ 32374 ] {} <Debug> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): There are no data parts
2026.05.06 08:48:35.488811 [ 32374 ] {} <Information> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.488850 [ 32351 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loading Active part 19732_10_10_0 from disk default
2026.05.06 08:48:35.488904 [ 32377 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loading Active part 19723_1_1_0 from disk default
2026.05.06 08:48:35.489044 [ 32388 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loading Active part 19726_4_4_0 from disk default
2026.05.06 08:48:35.489121 [ 32374 ] {} <Debug> DatabaseAtomic (shard_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.489015 [ 32394 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loading Active part 19727_5_5_0 from disk default
2026.05.06 08:48:35.489168 [ 32374 ] {} <Debug> DatabaseAtomic (shard_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.489110 [ 32337 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loading Active part 19725_3_3_0 from disk default
2026.05.06 08:48:35.489297 [ 32374 ] {} <Test> DatabaseAtomic (shard_1): Counting attached table from_0 to database shard_1
2026.05.06 08:48:35.489479 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.dest_view_stripe_log' with status OK
2026.05.06 08:48:35.489632 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.test_table' in BackgroundLoad
2026.05.06 08:48:35.489727 [ 32371 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.test_table
2026.05.06 08:48:35.489743 [ 32353 ] {} <Debug> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Loading data parts
2026.05.06 08:48:35.489828 [ 32398 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.dst' with status OK
2026.05.06 08:48:35.489895 [ 32398 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.t_proj_external' in BackgroundLoad
2026.05.06 08:48:35.489942 [ 32398 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.t_proj_external
2026.05.06 08:48:35.490157 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table shard_1.from_0' with status OK
2026.05.06 08:48:35.490401 [ 32390 ] {} <Debug> test_1.agg (44d18def-d554-419b-9256-9fca07aef37a): Loading data parts
2026.05.06 08:48:35.490522 [ 32378 ] {} <Test> test_13.collapsing_suspicious_granularity (3f575314-61b2-4c2a-87df-ac628b3fa131): loadDataPart: inserting all_1_1_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.490644 [ 32378 ] {} <Trace> test_13.collapsing_suspicious_granularity (3f575314-61b2-4c2a-87df-ac628b3fa131): Finished loading Active part all_1_1_1 on disk default
2026.05.06 08:48:35.496616 [ 32371 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.497327 [ 32353 ] {} <Test> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.497435 [ 32353 ] {} <Debug> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): There are no data parts
2026.05.06 08:48:35.499267 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.500071 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.dest_view_memory' in BackgroundLoad
2026.05.06 08:48:35.500166 [ 32371 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.500201 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.500206 [ 32353 ] {} <Information> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.500272 [ 32397 ] {} <Test> DatabaseAtomic (test_1): Counting attached table data to database test_1
2026.05.06 08:48:35.500257 [ 32374 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.dest_view_memory
2026.05.06 08:48:35.500298 [ 32371 ] {} <Test> DatabaseAtomic (test_15): Counting attached table test_table to database test_15
2026.05.06 08:48:35.500397 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.data' with status OK
2026.05.06 08:48:35.501398 [ 32353 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.500217 [ 32370 ] {} <Debug> test_13.collapsing_suspicious_granularity (3f575314-61b2-4c2a-87df-ac628b3fa131): Loaded data parts (1 items)
2026.05.06 08:48:35.502256 [ 32374 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.503052 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.table_dst' in BackgroundLoad
2026.05.06 08:48:35.503112 [ 32342 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loading Active part 19729_7_7_0 from disk default
2026.05.06 08:48:35.503153 [ 32397 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.table_dst
2026.05.06 08:48:35.503217 [ 32374 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.503238 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.test_table' with status OK
2026.05.06 08:48:35.503336 [ 32374 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table dest_view_memory to database test_truncate_database
2026.05.06 08:48:35.503930 [ 32370 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.504033 [ 32353 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.505010 [ 32370 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.505114 [ 32370 ] {} <Test> DatabaseAtomic (test_13): Counting attached table collapsing_suspicious_granularity to database test_13
2026.05.06 08:48:35.505242 [ 32353 ] {} <Test> DatabaseAtomic (test_15): Counting attached table t_r_ok to database test_15
2026.05.06 08:48:35.506579 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table shard_0.from_0' in BackgroundLoad
2026.05.06 08:48:35.506718 [ 32371 ] {} <Trace> DatabaseAtomic (shard_0): Loading table shard_0.from_0
2026.05.06 08:48:35.506824 [ 32390 ] {} <Test> test_1.agg (44d18def-d554-419b-9256-9fca07aef37a): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.506974 [ 32390 ] {} <Debug> test_1.agg (44d18def-d554-419b-9256-9fca07aef37a): There are no data parts
2026.05.06 08:48:35.507330 [ 32340 ] {} <Debug> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Loading data parts
2026.05.06 08:48:35.507764 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.dest_view_memory' with status OK
2026.05.06 08:48:35.507788 [ 32390 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.507858 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.source_table_memory' in BackgroundLoad
2026.05.06 08:48:35.507893 [ 32390 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.508330 [ 32340 ] {} <Test> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.508387 [ 32340 ] {} <Debug> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): There are no data parts
2026.05.06 08:48:35.509649 [ 32376 ] {} <Test> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): loadDataPart: inserting 19731_9_9_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.509723 [ 32376 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Finished loading Active part 19731_9_9_0 on disk default
2026.05.06 08:48:35.509737 [ 32377 ] {} <Test> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): loadDataPart: inserting 19723_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.509805 [ 32377 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Finished loading Active part 19723_1_1_0 on disk default
2026.05.06 08:48:35.509989 [ 32374 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.source_table_memory
2026.05.06 08:48:35.510039 [ 32390 ] {} <Test> DatabaseAtomic (test_1): Counting attached table agg to database test_1
2026.05.06 08:48:35.510097 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.collapsing_suspicious_granularity' with status OK
2026.05.06 08:48:35.510128 [ 32340 ] {} <Information> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.510181 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_01516.primary_key_test' in BackgroundLoad
2026.05.06 08:48:35.510235 [ 32370 ] {} <Trace> DatabaseOrdinary (test_01516): Loading table test_01516.primary_key_test
2026.05.06 08:48:35.510378 [ 32374 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.510427 [ 32374 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.510462 [ 32353 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.t_r_ok' with status OK
2026.05.06 08:48:35.510507 [ 32387 ] {} <Test> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): loadDataPart: inserting 19724_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.510640 [ 32365 ] {} <Test> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): loadDataPart: inserting 19728_6_6_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.510654 [ 32387 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Finished loading Active part 19724_2_2_0 on disk default
2026.05.06 08:48:35.510705 [ 32365 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Finished loading Active part 19728_6_6_0 on disk default
2026.05.06 08:48:35.510726 [ 32353 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.partslost_1' in BackgroundLoad
2026.05.06 08:48:35.510761 [ 32351 ] {} <Test> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): loadDataPart: inserting 19732_10_10_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.510533 [ 32340 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.510881 [ 32351 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Finished loading Active part 19732_10_10_0 on disk default
2026.05.06 08:48:35.510889 [ 32340 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.510945 [ 32353 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.partslost_1
2026.05.06 08:48:35.511012 [ 32340 ] {} <Test> DatabaseAtomic (test_15): Counting attached table t to database test_15
2026.05.06 08:48:35.511059 [ 32344 ] {} <Test> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): loadDataPart: inserting 19730_8_8_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.510500 [ 32374 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table source_table_memory to database test_truncate_database
2026.05.06 08:48:35.511189 [ 32344 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Finished loading Active part 19730_8_8_0 on disk default
2026.05.06 08:48:35.511194 [ 32342 ] {} <Test> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): loadDataPart: inserting 19729_7_7_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.511266 [ 32340 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.t' with status OK
2026.05.06 08:48:35.511362 [ 32342 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Finished loading Active part 19729_7_7_0 on disk default
2026.05.06 08:48:35.511371 [ 32394 ] {} <Test> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): loadDataPart: inserting 19727_5_5_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.511553 [ 32394 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Finished loading Active part 19727_5_5_0 on disk default
2026.05.06 08:48:35.511561 [ 32340 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.merge_tree' in BackgroundLoad
2026.05.06 08:48:35.511678 [ 32388 ] {} <Test> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): loadDataPart: inserting 19726_4_4_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.511716 [ 32340 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.merge_tree
2026.05.06 08:48:35.511772 [ 32388 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Finished loading Active part 19726_4_4_0 on disk default
2026.05.06 08:48:35.511846 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.source_table_memory' with status OK
2026.05.06 08:48:35.511974 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.test_s3_mt_dst' in BackgroundLoad
2026.05.06 08:48:35.512059 [ 32374 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.test_s3_mt_dst
2026.05.06 08:48:35.512524 [ 32395 ] {} <Trace> S3Client: Provider type: Unknown
2026.05.06 08:48:35.512726 [ 32395 ] {} <Trace> S3Client: API mode of the S3 client: AWS
2026.05.06 08:48:35.513438 [ 32390 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.agg' with status OK
2026.05.06 08:48:35.513512 [ 32390 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.table_gcd_codec_one_hundred_zeros' in BackgroundLoad
2026.05.06 08:48:35.513551 [ 32390 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.table_gcd_codec_one_hundred_zeros
2026.05.06 08:48:35.514210 [ 32337 ] {} <Test> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): loadDataPart: inserting 19725_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.514259 [ 32337 ] {} <Trace> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Finished loading Active part 19725_3_3_0 on disk default
2026.05.06 08:48:35.515387 [ 32381 ] {} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Loading data parts
2026.05.06 08:48:35.515741 [ 32401 ] {} <Debug> test_13.test_tuple_filter__fuzz_2 (88e5e6b9-eb4c-4a47-9539-05c6c61b60b7): Loaded data parts (10 items)
2026.05.06 08:48:35.516116 [ 32381 ] {} <Test> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.516433 [ 32401 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.516491 [ 32401 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.516543 [ 32401 ] {} <Test> DatabaseAtomic (test_13): Counting attached table test_tuple_filter__fuzz_2 to database test_13
2026.05.06 08:48:35.516615 [ 32339 ] {} <Debug> test_1.t2 (7bdcf49f-d945-4d28-b727-165c83183655): Loading data parts
2026.05.06 08:48:35.516679 [ 32401 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.test_tuple_filter__fuzz_2' with status OK
2026.05.06 08:48:35.517997 [ 32397 ] {} <Debug> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading data parts
2026.05.06 08:48:35.518359 [ 32381 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Will load 3 parts using up to 64 threads
2026.05.06 08:48:35.518588 [ 32401 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.t_index_hypothesis' in BackgroundLoad
2026.05.06 08:48:35.518635 [ 32401 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.t_index_hypothesis
2026.05.06 08:48:35.518946 [ 32353 ] {} <Debug> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Loading data parts
2026.05.06 08:48:35.519266 [ 32370 ] {} <Debug> test_01516.primary_key_test: Loading data parts
2026.05.06 08:48:35.519309 [ 32353 ] {} <Test> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.519425 [ 32397 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.519689 [ 32397 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Will load 16 parts using up to 64 threads
2026.05.06 08:48:35.519682 [ 32353 ] {} <Trace> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.519723 [ 32342 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Loading Active part all_3_3_0 from disk default
2026.05.06 08:48:35.519932 [ 32344 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Loading Active part all_0_0_0 from disk default
2026.05.06 08:48:35.520007 [ 32388 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.520117 [ 32370 ] {} <Test> test_01516.primary_key_test: loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.520164 [ 32370 ] {} <Trace> test_01516.primary_key_test: Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.520173 [ 32351 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_16_16_0 from disk default
2026.05.06 08:48:35.520242 [ 32377 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_12_12_0 from disk default
2026.05.06 08:48:35.520924 [ 32395 ] {} <Debug> DatabaseAtomic (test_mkf6rch8): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.521036 [ 32395 ] {} <Debug> DatabaseAtomic (test_mkf6rch8): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.521064 [ 32376 ] {} <Trace> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Loading Active part all_0_0_0_2 from disk default
2026.05.06 08:48:35.521152 [ 32395 ] {} <Test> DatabaseAtomic (test_mkf6rch8): Counting attached table test_02481_mismatch_files to database test_mkf6rch8
2026.05.06 08:48:35.521444 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_mkf6rch8.test_02481_mismatch_files' with status OK
2026.05.06 08:48:35.521644 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.mv_b_even_even' in BackgroundLoad
2026.05.06 08:48:35.521830 [ 32395 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.mv_b_even_even
2026.05.06 08:48:35.522056 [ 32342 ] {} <Test> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): loadDataPart: inserting all_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.522109 [ 32342 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Finished loading Active part all_3_3_0 on disk default
2026.05.06 08:48:35.522121 [ 32344 ] {} <Test> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): loadDataPart: inserting all_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.522190 [ 32344 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Finished loading Active part all_0_0_0 on disk default
2026.05.06 08:48:35.522279 [ 32342 ] {} <Trace> test_01516.primary_key_test: Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.522311 [ 32337 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_6_6_0 from disk default
2026.05.06 08:48:35.522395 [ 32402 ] {} <Debug> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Loading data parts
2026.05.06 08:48:35.522468 [ 32394 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.522620 [ 32343 ] {} <Debug> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Loading data parts
2026.05.06 08:48:35.522984 [ 32377 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_12_12_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.523049 [ 32377 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_12_12_0 on disk default
2026.05.06 08:48:35.523231 [ 32371 ] {} <Debug> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Loading data parts
2026.05.06 08:48:35.523276 [ 32351 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_16_16_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.523389 [ 32351 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_16_16_0 on disk default
2026.05.06 08:48:35.523707 [ 32377 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_5_5_0 from disk default
2026.05.06 08:48:35.523738 [ 32337 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_6_6_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.523800 [ 32337 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_6_6_0 on disk default
2026.05.06 08:48:35.523874 [ 32378 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.523957 [ 32339 ] {} <Test> test_1.t2 (7bdcf49f-d945-4d28-b727-165c83183655): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.523975 [ 32344 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_8_8_0 from disk default
2026.05.06 08:48:35.524018 [ 32339 ] {} <Trace> test_1.t2 (7bdcf49f-d945-4d28-b727-165c83183655): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.524035 [ 32390 ] {} <Debug> test_1.table_gcd_codec_one_hundred_zeros (7a8d8058-d62a-4aeb-906e-0e30867458e7): Loading data parts
2026.05.06 08:48:35.524078 [ 32342 ] {} <Test> test_01516.primary_key_test: loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.524127 [ 32342 ] {} <Trace> test_01516.primary_key_test: Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.524220 [ 32370 ] {} <Debug> test_01516.primary_key_test: Loaded data parts (1 items)
2026.05.06 08:48:35.524329 [ 32388 ] {} <Test> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.524498 [ 32388 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.524577 [ 32371 ] {} <Test> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.524612 [ 32343 ] {} <Test> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.524632 [ 32371 ] {} <Debug> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): There are no data parts
2026.05.06 08:48:35.524662 [ 32343 ] {} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Will load 4 parts using up to 64 threads
2026.05.06 08:48:35.524758 [ 32418 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_15_15_0 from disk default
2026.05.06 08:48:35.525042 [ 32402 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.525159 [ 32402 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Will load 4 parts using up to 64 threads
2026.05.06 08:48:35.525273 [ 32371 ] {} <Information> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.525439 [ 32342 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Loading Active part 2_0_0_0 from disk default
2026.05.06 08:48:35.525464 [ 32365 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Loading Active part 1_3_3_0 from disk default
2026.05.06 08:48:35.525549 [ 32388 ] {} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Loading Active part all_3_3_0 from disk default
2026.05.06 08:48:35.525561 [ 32387 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Loading Active part 0_0_0_0 from disk default
2026.05.06 08:48:35.525600 [ 32351 ] {} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.525725 [ 32371 ] {} <Debug> DatabaseAtomic (shard_0): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.525931 [ 32377 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_5_5_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.525981 [ 32377 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_5_5_0 on disk default
2026.05.06 08:48:35.526276 [ 32371 ] {} <Debug> DatabaseAtomic (shard_0): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.526342 [ 32371 ] {} <Test> DatabaseAtomic (shard_0): Counting attached table from_0 to database shard_0
2026.05.06 08:48:35.526345 [ 32394 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.526359 [ 32390 ] {} <Test> test_1.table_gcd_codec_one_hundred_zeros (7a8d8058-d62a-4aeb-906e-0e30867458e7): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.526503 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table shard_0.from_0' with status OK
2026.05.06 08:48:35.526508 [ 32394 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.526580 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.test_table_comment' in BackgroundLoad
2026.05.06 08:48:35.526635 [ 32371 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.test_table_comment
2026.05.06 08:48:35.526735 [ 32403 ] {} <Debug> test.`.inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8` (e093710e-4a24-49d0-b8f0-d54ae5f07bae): Loading data parts
2026.05.06 08:48:35.527095 [ 32403 ] {} <Test> test.`.inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8` (e093710e-4a24-49d0-b8f0-d54ae5f07bae): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.527179 [ 32403 ] {} <Debug> test.`.inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8` (e093710e-4a24-49d0-b8f0-d54ae5f07bae): There are no data parts
2026.05.06 08:48:35.526148 [ 32370 ] {} <Test> DatabaseOrdinary (test_01516): Counting attached table primary_key_test to database test_01516
2026.05.06 08:48:35.529165 [ 32403 ] {} <Debug> DatabaseAtomic (test): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.526459 [ 32390 ] {} <Debug> test_1.table_gcd_codec_one_hundred_zeros (7a8d8058-d62a-4aeb-906e-0e30867458e7): There are no data parts
2026.05.06 08:48:35.528006 [ 32376 ] {} <Test> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): loadDataPart: inserting all_0_0_0_2 (state Active) into data_parts_indexes
2026.05.06 08:48:35.528051 [ 32344 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_8_8_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.529319 [ 32377 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Loading Active part 1_4_4_0 from disk default
2026.05.06 08:48:35.529329 [ 32376 ] {} <Trace> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Finished loading Active part all_0_0_0_2 on disk default
2026.05.06 08:48:35.525018 [ 32421 ] {} <Trace> test_1.t2 (7bdcf49f-d945-4d28-b727-165c83183655): Loading Active part 2_1_1_0 from disk default
2026.05.06 08:48:35.529405 [ 32344 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_8_8_0 on disk default
2026.05.06 08:48:35.529448 [ 32378 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.529585 [ 32378 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.531160 [ 32388 ] {} <Test> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): loadDataPart: inserting all_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.531211 [ 32388 ] {} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Finished loading Active part all_3_3_0 on disk default
2026.05.06 08:48:35.531952 [ 32403 ] {} <Debug> DatabaseAtomic (test): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.532138 [ 32403 ] {} <Test> DatabaseAtomic (test): Counting attached table .inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8 to database test
2026.05.06 08:48:35.532521 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_01516.primary_key_test' with status OK
2026.05.06 08:48:35.532730 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1..inner_id.8ca14fb2-e48e-4ec8-8fdf-854e1dc4cf8c' in BackgroundLoad
2026.05.06 08:48:35.532799 [ 32370 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1..inner_id.8ca14fb2-e48e-4ec8-8fdf-854e1dc4cf8c
2026.05.06 08:48:35.533009 [ 32403 ] {} <Debug> AsyncLoader: Finish load job 'load table test..inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8' with status OK
2026.05.06 08:48:35.533111 [ 32403 ] {} <Debug> AsyncLoader: Execute load job 'load table shard_0.data_02346' in BackgroundLoad
2026.05.06 08:48:35.533172 [ 32403 ] {} <Trace> DatabaseAtomic (shard_0): Loading table shard_0.data_02346
2026.05.06 08:48:35.533249 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk default
2026.05.06 08:48:35.533256 [ 32381 ] {} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Loaded data parts (3 items)
2026.05.06 08:48:35.533257 [ 32353 ] {} <Debug> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Loaded data parts (1 items)
2026.05.06 08:48:35.533580 [ 32351 ] {} <Test> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.533613 [ 32353 ] {} <Information> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.533638 [ 32351 ] {} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.533661 [ 32370 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.533827 [ 32370 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.533944 [ 32353 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.533987 [ 32370 ] {} <Test> DatabaseAtomic (test_1): Counting attached table .inner_id.8ca14fb2-e48e-4ec8-8fdf-854e1dc4cf8c to database test_1
2026.05.06 08:48:35.534011 [ 32353 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.534080 [ 32353 ] {} <Test> DatabaseAtomic (test_15): Counting attached table partslost_1 to database test_15
2026.05.06 08:48:35.534110 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1..inner_id.8ca14fb2-e48e-4ec8-8fdf-854e1dc4cf8c' with status OK
2026.05.06 08:48:35.533299 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk disk1_02961
2026.05.06 08:48:35.534194 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.input_function_table_3' in BackgroundLoad
2026.05.06 08:48:35.534269 [ 32370 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.input_function_table_3
2026.05.06 08:48:35.534313 [ 32418 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_15_15_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.534371 [ 32418 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_15_15_0 on disk default
2026.05.06 08:48:35.534397 [ 32353 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.partslost_1' with status OK
2026.05.06 08:48:35.534456 [ 32365 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): loadDataPart: inserting 1_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.534505 [ 32353 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.test_s3_mt' in BackgroundLoad
2026.05.06 08:48:35.534534 [ 32365 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Finished loading Active part 1_3_3_0 on disk default
2026.05.06 08:48:35.534618 [ 32353 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.test_s3_mt
2026.05.06 08:48:35.533594 [ 32423 ] {} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Loading Active part all_4_4_0 from disk default
2026.05.06 08:48:35.534761 [ 32390 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.534803 [ 32390 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.534817 [ 32387 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): loadDataPart: inserting 0_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.534855 [ 32390 ] {} <Test> DatabaseAtomic (test_1): Counting attached table table_gcd_codec_one_hundred_zeros to database test_1
2026.05.06 08:48:35.527689 [ 32419 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_11_11_0 from disk default
2026.05.06 08:48:35.534897 [ 32387 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Finished loading Active part 0_0_0_0 on disk default
2026.05.06 08:48:35.535000 [ 32390 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.table_gcd_codec_one_hundred_zeros' with status OK
2026.05.06 08:48:35.535054 [ 32418 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_14_14_0 from disk default
2026.05.06 08:48:35.535099 [ 32351 ] {} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.535121 [ 32390 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.test' in BackgroundLoad
2026.05.06 08:48:35.535134 [ 32340 ] {} <Debug> test_15.merge_tree (ae66fd0f-b563-4dde-aae5-c8507d5f5db3): Loading data parts
2026.05.06 08:48:35.535202 [ 32388 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_7_7_0 from disk default
2026.05.06 08:48:35.535257 [ 32378 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_10_10_0 from disk default
2026.05.06 08:48:35.534197 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk disk2_02961
2026.05.06 08:48:35.535418 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk disk3_02961
2026.05.06 08:48:35.535451 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk disk4_02961
2026.05.06 08:48:35.535500 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_cache
2026.05.06 08:48:35.535524 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_cache_2
2026.05.06 08:48:35.535549 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_cache_3
2026.05.06 08:48:35.535571 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_disk
2026.05.06 08:48:35.535566 [ 32377 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): loadDataPart: inserting 1_4_4_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.535643 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_disk_2
2026.05.06 08:48:35.535670 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_disk_3
2026.05.06 08:48:35.535701 [ 32377 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Finished loading Active part 1_4_4_0 on disk default
2026.05.06 08:48:35.535794 [ 32365 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_4_4_0 from disk default
2026.05.06 08:48:35.535892 [ 32344 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_13_13_0 from disk default
2026.05.06 08:48:35.535976 [ 32390 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.test
2026.05.06 08:48:35.535968 [ 32394 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_9_9_0 from disk default
2026.05.06 08:48:35.536017 [ 32337 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loading Active part all_3_3_0 from disk default
2026.05.06 08:48:35.536075 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.535908 [ 32342 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): loadDataPart: inserting 2_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.536147 [ 32381 ] {} <Information> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.536286 [ 32342 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Finished loading Active part 2_0_0_0 on disk default
2026.05.06 08:48:35.536320 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.536401 [ 32371 ] {} <Test> DatabaseAtomic (test_13): Counting attached table test_table_comment to database test_13
2026.05.06 08:48:35.536592 [ 32370 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.536695 [ 32370 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.536794 [ 32370 ] {} <Test> DatabaseAtomic (test_1): Counting attached table input_function_table_3 to database test_1
2026.05.06 08:48:35.538007 [ 32344 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_13_13_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.538059 [ 32344 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_13_13_0 on disk default
2026.05.06 08:48:35.538394 [ 32418 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_14_14_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.538443 [ 32418 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_14_14_0 on disk default
2026.05.06 08:48:35.538533 [ 32402 ] {} <Debug> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Loaded data parts (4 items)
2026.05.06 08:48:35.538615 [ 32402 ] {} <Information> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.538947 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.test_table_comment' with status OK
2026.05.06 08:48:35.538990 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.539148 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.539184 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.mv' in BackgroundLoad
2026.05.06 08:48:35.539198 [ 32381 ] {} <Test> DatabaseAtomic (test_1): Counting attached table test_without_merge to database test_1
2026.05.06 08:48:35.539242 [ 32371 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.mv
2026.05.06 08:48:35.539312 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.test_without_merge' with status OK
2026.05.06 08:48:35.539366 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.array_jaccard_index' in BackgroundLoad
2026.05.06 08:48:35.539408 [ 32381 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.array_jaccard_index
2026.05.06 08:48:35.539473 [ 32402 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.539536 [ 32402 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.539546 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.input_function_table_3' with status OK
2026.05.06 08:48:35.539799 [ 32337 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.539852 [ 32337 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_3_3_0 on disk default
2026.05.06 08:48:35.539914 [ 32388 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_7_7_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.539995 [ 32388 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_7_7_0 on disk default
2026.05.06 08:48:35.540053 [ 32394 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_9_9_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.540097 [ 32394 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_9_9_0 on disk default
2026.05.06 08:48:35.540132 [ 32365 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_4_4_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.540179 [ 32365 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_4_4_0 on disk default
2026.05.06 08:48:35.540231 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.540724 [ 32351 ] {} <Test> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.540770 [ 32351 ] {} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.541621 [ 32378 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_10_10_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.541734 [ 32378 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_10_10_0 on disk default
2026.05.06 08:48:35.542719 [ 32402 ] {} <Test> DatabaseAtomic (test_15): Counting attached table src to database test_15
2026.05.06 08:48:35.542817 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.dest_view_merge_tree' in BackgroundLoad
2026.05.06 08:48:35.542838 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.542975 [ 32370 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.dest_view_merge_tree
2026.05.06 08:48:35.543153 [ 32402 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.src' with status OK
2026.05.06 08:48:35.543428 [ 32371 ] {} <Test> DatabaseAtomic (test_13): Counting attached table mv to database test_13
2026.05.06 08:48:35.543520 [ 32340 ] {} <Test> test_15.merge_tree (ae66fd0f-b563-4dde-aae5-c8507d5f5db3): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.543639 [ 32340 ] {} <Trace> test_15.merge_tree (ae66fd0f-b563-4dde-aae5-c8507d5f5db3): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.543679 [ 32402 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.numbers_squashed' in BackgroundLoad
2026.05.06 08:48:35.543739 [ 32402 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.numbers_squashed
2026.05.06 08:48:35.543893 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_cache
2026.05.06 08:48:35.543943 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.544012 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.544073 [ 32395 ] {} <Test> DatabaseAtomic (test_1): Counting attached table mv_b_even_even to database test_1
2026.05.06 08:48:35.544104 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_cache_02933
2026.05.06 08:48:35.544274 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.mv_b_even_even' with status OK
2026.05.06 08:48:35.544281 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_cache_02944
2026.05.06 08:48:35.544317 [ 32370 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.544385 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.A' in BackgroundLoad
2026.05.06 08:48:35.544477 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_cache_multi
2026.05.06 08:48:35.544562 [ 32370 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.544615 [ 32395 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.A
2026.05.06 08:48:35.544665 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_cache_multi_2
2026.05.06 08:48:35.544698 [ 32370 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table dest_view_merge_tree to database test_truncate_database
2026.05.06 08:48:35.544836 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_disk
2026.05.06 08:48:35.544864 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_disk_02944
2026.05.06 08:48:35.544888 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_disk_02963
2026.05.06 08:48:35.544912 [ 31798 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_no_cache
2026.05.06 08:48:35.544974 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.dest_view_merge_tree' with status OK
2026.05.06 08:48:35.545110 [ 32402 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.545132 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.json_test' in BackgroundLoad
2026.05.06 08:48:35.545152 [ 32402 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.545230 [ 32402 ] {} <Test> DatabaseAtomic (test_13): Counting attached table numbers_squashed to database test_13
2026.05.06 08:48:35.545270 [ 32370 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.json_test
2026.05.06 08:48:35.545294 [ 32395 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.545319 [ 32402 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.numbers_squashed' with status OK
2026.05.06 08:48:35.545357 [ 32395 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.545918 [ 32402 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.table_for_freeze' in BackgroundLoad
2026.05.06 08:48:35.545987 [ 32402 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.table_for_freeze
2026.05.06 08:48:35.545992 [ 32395 ] {} <Test> DatabaseAtomic (test_13): Counting attached table A to database test_13
2026.05.06 08:48:35.546105 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.A' with status OK
2026.05.06 08:48:35.546195 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.table_when_b_even' in BackgroundLoad
2026.05.06 08:48:35.546253 [ 32395 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.table_when_b_even
2026.05.06 08:48:35.546723 [ 32368 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.546770 [ 32368 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.546812 [ 32368 ] {} <Test> DatabaseAtomic (test_13): Counting attached table constrained to database test_13
2026.05.06 08:48:35.546948 [ 32368 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.constrained' with status OK
2026.05.06 08:48:35.547002 [ 32368 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.B' in BackgroundLoad
2026.05.06 08:48:35.547077 [ 32368 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.B
2026.05.06 08:48:35.547252 [ 32421 ] {} <Test> test_1.t2 (7bdcf49f-d945-4d28-b727-165c83183655): loadDataPart: inserting 2_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.547342 [ 32421 ] {} <Trace> test_1.t2 (7bdcf49f-d945-4d28-b727-165c83183655): Finished loading Active part 2_1_1_0 on disk default
2026.05.06 08:48:35.547505 [ 32339 ] {} <Debug> test_1.t2 (7bdcf49f-d945-4d28-b727-165c83183655): Loaded data parts (1 items)
2026.05.06 08:48:35.547562 [ 32421 ] {} <Trace> test_15.merge_tree (ae66fd0f-b563-4dde-aae5-c8507d5f5db3): Loading Active part 20000101_20000101_1_1_0 from disk default
2026.05.06 08:48:35.547580 [ 32419 ] {} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): loadDataPart: inserting all_11_11_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.547735 [ 32419 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Finished loading Active part all_11_11_0 on disk default
2026.05.06 08:48:35.547787 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.mv' with status OK
2026.05.06 08:48:35.547875 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.source_table_merge_tree' in BackgroundLoad
2026.05.06 08:48:35.547931 [ 32423 ] {} <Test> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): loadDataPart: inserting all_4_4_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.547933 [ 32371 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.source_table_merge_tree
2026.05.06 08:48:35.547981 [ 32397 ] {} <Debug> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Loaded data parts (16 items)
2026.05.06 08:48:35.548016 [ 32423 ] {} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Finished loading Active part all_4_4_0 on disk default
2026.05.06 08:48:35.548315 [ 32343 ] {} <Debug> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Loaded data parts (4 items)
2026.05.06 08:48:35.550490 [ 32374 ] {} <Information> AWSClient: ClientConfiguration: User agent is overridden in the config: ClickHouse 24.8.14.10546.altinitytest (altinity build)
2026.05.06 08:48:35.550855 [ 32421 ] {} <Test> test_15.merge_tree (ae66fd0f-b563-4dde-aae5-c8507d5f5db3): loadDataPart: inserting 20000101_20000101_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.550939 [ 32421 ] {} <Trace> test_15.merge_tree (ae66fd0f-b563-4dde-aae5-c8507d5f5db3): Finished loading Active part 20000101_20000101_1_1_0 on disk default
2026.05.06 08:48:35.551106 [ 32340 ] {} <Debug> test_15.merge_tree (ae66fd0f-b563-4dde-aae5-c8507d5f5db3): Loaded data parts (1 items)
2026.05.06 08:48:35.553346 [ 31633 ] {} <Debug> CertificateReloader: Reloading certificate (/etc/clickhouse-server/server.crt) and key (/etc/clickhouse-server/server.key).
2026.05.06 08:48:35.554618 [ 31633 ] {} <Information> CertificateReloader: Reloaded certificate (/etc/clickhouse-server/server.crt) and key (/etc/clickhouse-server/server.key).
2026.05.06 08:48:35.554698 [ 31633 ] {} <Debug> CertificateReloader: Initializing certificate reloader.
2026.05.06 08:48:35.554895 [ 31633 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 30.79 MiB, peak 59.86 MiB, free memory in arenas 0.00 B, will set to 2.09 GiB (RSS), difference: 2.06 GiB
2026.05.06 08:48:35.555004 [ 31633 ] {} <Debug> MemoryTracker: Current memory usage (total): 2.09 GiB.
2026.05.06 08:48:35.559254 [ 32374 ] {} <Trace> S3Client: Provider type: Unknown
2026.05.06 08:48:35.559364 [ 32374 ] {} <Trace> S3Client: API mode of the S3 client: AWS
2026.05.06 08:48:35.583106 [ 32374 ] {} <Debug> S3ObjectStorage: Objects with paths [03008_test_s3_mt/clickhouse_remove_objects_capability_11a20655-72d2-4e0e-b157-3c8461d3c947] were removed from S3
2026.05.06 08:48:35.586916 [ 32374 ] {} <Debug> MetadataStorageFromPlainObjectStorage: Loading metadata
2026.05.06 08:48:35.603571 [ 32374 ] {} <Debug> MetadataStorageFromPlainObjectStorage: Loaded metadata for 5 files, found 5 directories
2026.05.06 08:48:35.603788 [ 32374 ] {} <Debug> MetadataStorageFromPlainObjectStorage: Loading directory tree
2026.05.06 08:48:35.609638 [ 32374 ] {} <Debug> MetadataStorageFromPlainObjectStorage: Loaded directory tree for 5 directories, found 19 files
2026.05.06 08:48:35.610009 [ 32374 ] {} <Test> DiskObjectStorage(03008_s3_plain_rewritable): Write file: clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:35.633205 [ 32374 ] {} <Test> MetadataStorageFromPlainObjectStorageWriteFileOperation: Creating metadata for a file  'clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947'
2026.05.06 08:48:35.633497 [ 32374 ] {} <Trace> MetadataStorageFromPlainObjectStorageWriteFileOperation: Parent dirrectory does not exist, skipping path clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:35.641332 [ 32374 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947 (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:35.654497 [ 32374 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947 (clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947)
2026.05.06 08:48:35.654718 [ 32374 ] {} <Test> ReadBufferFromS3: Read S3 object. Bucket: test, Key: 03008_test_s3_mt/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947, Version: Latest, Offset: 2
2026.05.06 08:48:35.677072 [ 32374 ] {} <Test> MetadataStorageFromPlainObjectStorageUnlinkMetadataFileOperation: Unlinking metadata for a write 'clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947' with remote path '03008_test_s3_mt/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947'
2026.05.06 08:48:35.677277 [ 32374 ] {} <Trace> MetadataStorageFromPlainObjectStorageUnlinkMetadataFileOperation: Parent directory does not exist, skipping path clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947
2026.05.06 08:48:35.682996 [ 32374 ] {} <Debug> S3ObjectStorage: Objects with paths [03008_test_s3_mt/clickhouse_access_check_11a20655-72d2-4e0e-b157-3c8461d3c947] were removed from S3
2026.05.06 08:48:35.683224 [ 32374 ] {} <Information> DiskObjectStorage(03008_s3_plain_rewritable): Starting up disk 03008_s3_plain_rewritable
2026.05.06 08:48:35.683368 [ 32374 ] {} <Information> DiskObjectStorage(03008_s3_plain_rewritable): Disk 03008_s3_plain_rewritable started up
2026.05.06 08:48:35.683956 [ 32403 ] {} <Debug> shard_0.data_02346 (493c9e05-7f03-45df-8dc4-abb4ed1db93f): Loading data parts
2026.05.06 08:48:35.684370 [ 32381 ] {} <Debug> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): Loading data parts
2026.05.06 08:48:35.684422 [ 32398 ] {} <Debug> test_15.t_proj_external (87447c1e-c29e-4e0e-af58-72cfc77edc4b): Loading data parts
2026.05.06 08:48:35.684522 [ 32368 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.684564 [ 32401 ] {} <Debug> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): Loading data parts
2026.05.06 08:48:35.684570 [ 32353 ] {} <Debug> MergeTreeSettings: Created custom disk 03008_s3_plain_rewritable
2026.05.06 08:48:35.684684 [ 32390 ] {} <Debug> test_13.test (40407d88-838f-4821-9923-ec6b735cb457): Loading data parts
2026.05.06 08:48:35.684569 [ 32374 ] {} <Debug> MergeTreeSettings: Created custom disk 03008_s3_plain_rewritable
2026.05.06 08:48:35.684713 [ 32371 ] {} <Debug> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loading data parts
2026.05.06 08:48:35.684581 [ 32402 ] {} <Debug> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading data parts
2026.05.06 08:48:35.684455 [ 32395 ] {} <Debug> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading data parts
2026.05.06 08:48:35.684654 [ 32368 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.684857 [ 32339 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.684697 [ 32403 ] {} <Test> shard_0.data_02346 (493c9e05-7f03-45df-8dc4-abb4ed1db93f): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.684752 [ 32370 ] {} <Debug> test_13.json_test (9048d3f2-57a4-4fab-9e7f-11419ec07f25): Loading data parts
2026.05.06 08:48:35.685020 [ 32403 ] {} <Trace> shard_0.data_02346 (493c9e05-7f03-45df-8dc4-abb4ed1db93f): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.685209 [ 32343 ] {} <Debug> DatabaseAtomic (test_w50hoezt): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.685304 [ 32343 ] {} <Debug> DatabaseAtomic (test_w50hoezt): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.685334 [ 32390 ] {} <Test> test_13.test (40407d88-838f-4821-9923-ec6b735cb457): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.685407 [ 32381 ] {} <Test> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.685469 [ 32421 ] {} <Trace> shard_0.data_02346 (493c9e05-7f03-45df-8dc4-abb4ed1db93f): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.685554 [ 32381 ] {} <Trace> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): Will load 3 parts using up to 64 threads
2026.05.06 08:48:35.685571 [ 32368 ] {} <Test> DatabaseAtomic (test_13): Counting attached table B to database test_13
2026.05.06 08:48:35.685599 [ 32339 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.685557 [ 32390 ] {} <Trace> test_13.test (40407d88-838f-4821-9923-ec6b735cb457): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.685369 [ 32340 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.685744 [ 32401 ] {} <Test> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.685785 [ 32368 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.B' with status OK
2026.05.06 08:48:35.685833 [ 32398 ] {} <Test> test_15.t_proj_external (87447c1e-c29e-4e0e-af58-72cfc77edc4b): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.685864 [ 32339 ] {} <Test> DatabaseAtomic (test_1): Counting attached table t2 to database test_1
2026.05.06 08:48:35.685849 [ 32340 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.685927 [ 32401 ] {} <Trace> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): Will load 4 parts using up to 64 threads
2026.05.06 08:48:35.685959 [ 32398 ] {} <Trace> test_15.t_proj_external (87447c1e-c29e-4e0e-af58-72cfc77edc4b): Will load 2 parts using up to 64 threads
2026.05.06 08:48:35.685968 [ 32368 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.table_for_rename_replicated' in BackgroundLoad
2026.05.06 08:48:35.686007 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.686064 [ 32377 ] {} <Trace> test_13.test (40407d88-838f-4821-9923-ec6b735cb457): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.686116 [ 32371 ] {} <Test> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.686087 [ 32368 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.table_for_rename_replicated
2026.05.06 08:48:35.685415 [ 32343 ] {} <Test> DatabaseAtomic (test_w50hoezt): Counting attached table test_constraints to database test_w50hoezt
2026.05.06 08:48:35.686021 [ 32340 ] {} <Test> DatabaseAtomic (test_15): Counting attached table merge_tree to database test_15
2026.05.06 08:48:35.686268 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.686253 [ 32316 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Found 43 old parts to remove. Parts: [202605_1_130_13, 202605_131_221_14, 202605_131_252_15, 202605_222_232_2, 202605_222_237_3, 202605_222_242_4, 202605_222_247_5, 202605_222_252_6, 202605_233_233_0, 202605_234_234_0, 202605_235_235_0, 202605_236_236_0, 202605_237_237_0, 202605_238_238_0, 202605_239_239_0, 202605_240_240_0, 202605_241_241_0, 202605_242_242_0, 202605_243_243_0, 202605_244_244_0, 202605_245_245_0, 202605_246_246_0, 202605_247_247_0, 202605_248_248_0, 202605_249_249_0, 202605_250_250_0, 202605_251_251_0, 202605_252_252_0, 202605_253_253_0, 202605_253_258_1, 202605_253_263_2, 202605_253_268_3, 202605_253_273_4, 202605_253_278_5, 202605_253_283_6, 202605_253_288_7, 202605_253_294_8, 202605_253_300_9, 202605_254_254_0, 202605_255_255_0, 202605_256_256_0, 202605_257_257_0, 202605_260_260_0]
2026.05.06 08:48:35.686335 [ 32371 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Will load 10 parts using up to 64 threads
2026.05.06 08:48:35.686215 [ 32402 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.686316 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.t2' with status OK
2026.05.06 08:48:35.686437 [ 32419 ] {} <Trace> test_15.t_proj_external (87447c1e-c29e-4e0e-af58-72cfc77edc4b): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.686405 [ 32370 ] {} <Test> test_13.json_test (9048d3f2-57a4-4fab-9e7f-11419ec07f25): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.686527 [ 32394 ] {} <Trace> test_15.t_proj_external (87447c1e-c29e-4e0e-af58-72cfc77edc4b): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.686552 [ 32397 ] {} <Test> DatabaseAtomic (test_1): Counting attached table table_dst to database test_1
2026.05.06 08:48:35.686509 [ 32402 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Will load 11 parts using up to 64 threads
2026.05.06 08:48:35.686614 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.test_table_2' in BackgroundLoad
2026.05.06 08:48:35.686573 [ 32395 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.686444 [ 32344 ] {} <Trace> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): Loading Active part all_9_9_0 from disk default
2026.05.06 08:48:35.686748 [ 32339 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.test_table_2
2026.05.06 08:48:35.686754 [ 32395 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Will load 20 parts using up to 64 threads
2026.05.06 08:48:35.686832 [ 32316 ] {} <Debug> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Removing 43 parts from filesystem (serially): Parts: [202605_1_130_13, 202605_131_221_14, 202605_131_252_15, 202605_222_232_2, 202605_222_237_3, 202605_222_242_4, 202605_222_247_5, 202605_222_252_6, 202605_233_233_0, 202605_234_234_0, 202605_235_235_0, 202605_236_236_0, 202605_237_237_0, 202605_238_238_0, 202605_239_239_0, 202605_240_240_0, 202605_241_241_0, 202605_242_242_0, 202605_243_243_0, 202605_244_244_0, 202605_245_245_0, 202605_246_246_0, 202605_247_247_0, 202605_248_248_0, 202605_249_249_0, 202605_250_250_0, 202605_251_251_0, 202605_252_252_0, 202605_253_253_0, 202605_253_258_1, 202605_253_263_2, 202605_253_268_3, 202605_253_273_4, 202605_253_278_5, 202605_253_283_6, 202605_253_288_7, 202605_253_294_8, 202605_253_300_9, 202605_254_254_0, 202605_255_255_0, 202605_256_256_0, 202605_257_257_0, 202605_260_260_0]
2026.05.06 08:48:35.686733 [ 32422 ] {} <Trace> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): Loading Active part all_7_7_0 from disk default
2026.05.06 08:48:35.686966 [ 32343 ] {} <Debug> AsyncLoader: Finish load job 'load table test_w50hoezt.test_constraints' with status OK
2026.05.06 08:48:35.686990 [ 32351 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading Active part 0_1_1_0 from disk default
2026.05.06 08:48:35.687095 [ 32423 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading Active part 8_9_9_0 from disk default
2026.05.06 08:48:35.687114 [ 32343 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.test1' in BackgroundLoad
2026.05.06 08:48:35.687214 [ 32343 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.test1
2026.05.06 08:48:35.688133 [ 32425 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading Active part 3_11_11_0 from disk default
2026.05.06 08:48:35.688453 [ 32374 ] {} <Trace> StoragePolicy (__03008_s3_plain_rewritable): Storage policy __03008_s3_plain_rewritable created, total volumes 1
2026.05.06 08:48:35.688547 [ 32424 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading Active part 1_2_2_0 from disk default
2026.05.06 08:48:35.688913 [ 32337 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading Active part 2_3_3_0 from disk default
2026.05.06 08:48:35.688932 [ 32388 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading Active part 7_8_8_0 from disk default
2026.05.06 08:48:35.686682 [ 32365 ] {} <Trace> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): Loading Active part all_8_8_0 from disk default
2026.05.06 08:48:35.686962 [ 32378 ] {} <Trace> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): Loading Active part all_1_6_1 from disk default
2026.05.06 08:48:35.690546 [ 32387 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading Active part 4_5_5_0 from disk default
2026.05.06 08:48:35.690641 [ 32376 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading Active part 5_13_13_0 from disk default
2026.05.06 08:48:35.690676 [ 32418 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading Active part 9_10_10_0 from disk default
2026.05.06 08:48:35.690692 [ 32342 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading Active part 3_12_12_0 from disk default
2026.05.06 08:48:35.690997 [ 32370 ] {} <Trace> test_13.json_test (9048d3f2-57a4-4fab-9e7f-11419ec07f25): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.691254 [ 32340 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.merge_tree' with status OK
2026.05.06 08:48:35.691291 [ 32421 ] {} <Test> shard_0.data_02346 (493c9e05-7f03-45df-8dc4-abb4ed1db93f): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.691370 [ 32421 ] {} <Trace> shard_0.data_02346 (493c9e05-7f03-45df-8dc4-abb4ed1db93f): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.693157 [ 32340 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.t0' in BackgroundLoad
2026.05.06 08:48:35.693269 [ 32340 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.t0
2026.05.06 08:48:35.693352 [ 32403 ] {} <Debug> shard_0.data_02346 (493c9e05-7f03-45df-8dc4-abb4ed1db93f): Loaded data parts (1 items)
2026.05.06 08:48:35.693544 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.table_dst' with status OK
2026.05.06 08:48:35.693694 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.ps' in BackgroundLoad
2026.05.06 08:48:35.693792 [ 32397 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.ps
2026.05.06 08:48:35.694419 [ 32403 ] {} <Debug> DatabaseAtomic (shard_0): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.697135 [ 32403 ] {} <Debug> DatabaseAtomic (shard_0): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.697202 [ 32403 ] {} <Test> DatabaseAtomic (shard_0): Counting attached table data_02346 to database shard_0
2026.05.06 08:48:35.697237 [ 32340 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.697314 [ 32340 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.697336 [ 32403 ] {} <Debug> AsyncLoader: Finish load job 'load table shard_0.data_02346' with status OK
2026.05.06 08:48:35.697393 [ 32340 ] {} <Test> DatabaseAtomic (test_1): Counting attached table t0 to database test_1
2026.05.06 08:48:35.697404 [ 32403 ] {} <Debug> AsyncLoader: Execute load job 'load table test_2gxskyku_1.t' in BackgroundLoad
2026.05.06 08:48:35.697434 [ 32397 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.697470 [ 32403 ] {} <Trace> DatabaseAtomic (test_2gxskyku_1): Loading table test_2gxskyku_1.t
2026.05.06 08:48:35.697475 [ 32397 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.697623 [ 32340 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.t0' with status OK
2026.05.06 08:48:35.697700 [ 32403 ] {} <Debug> DatabaseAtomic (test_2gxskyku_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.697939 [ 32397 ] {} <Test> DatabaseAtomic (test_15): Counting attached table ps to database test_15
2026.05.06 08:48:35.695943 [ 32426 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loading Active part 6_7_7_0 from disk default
2026.05.06 08:48:35.698399 [ 32340 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.test_collation' in BackgroundLoad
2026.05.06 08:48:35.698462 [ 32403 ] {} <Debug> DatabaseAtomic (test_2gxskyku_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.698572 [ 32340 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.test_collation
2026.05.06 08:48:35.698604 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.ps' with status OK
2026.05.06 08:48:35.698595 [ 32403 ] {} <Test> DatabaseAtomic (test_2gxskyku_1): Counting attached table t to database test_2gxskyku_1
2026.05.06 08:48:35.698691 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.rename_table_multiple' in BackgroundLoad
2026.05.06 08:48:35.698732 [ 32397 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.rename_table_multiple
2026.05.06 08:48:35.698874 [ 32403 ] {} <Debug> AsyncLoader: Finish load job 'load table test_2gxskyku_1.t' with status OK
2026.05.06 08:48:35.699799 [ 32421 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loading Active part 8_9_9_0 from disk default
2026.05.06 08:48:35.700841 [ 32394 ] {} <Test> test_15.t_proj_external (87447c1e-c29e-4e0e-af58-72cfc77edc4b): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.700964 [ 32394 ] {} <Trace> test_15.t_proj_external (87447c1e-c29e-4e0e-af58-72cfc77edc4b): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.701691 [ 32403 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.table_for_freeze_old_syntax' in BackgroundLoad
2026.05.06 08:48:35.702102 [ 32403 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.table_for_freeze_old_syntax
2026.05.06 08:48:35.702922 [ 32343 ] {} <Debug> test_1.test1 (5d763aa7-9b30-4a61-8f15-11f7ca5b4d88): Loading data parts
2026.05.06 08:48:35.706407 [ 32339 ] {} <Debug> test_13.test_table_2 (97e92dfd-a651-4d7a-94bb-959bc6bd2a6f): Loading data parts
2026.05.06 08:48:35.707125 [ 32394 ] {} <Trace> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.710576 [ 32340 ] {} <Debug> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Loading data parts
2026.05.06 08:48:35.714852 [ 32428 ] {} <Trace> test_13.json_test (9048d3f2-57a4-4fab-9e7f-11419ec07f25): Loading Active part all_1_1_0_2 from disk default
2026.05.06 08:48:35.716873 [ 32397 ] {} <Debug> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Loading data parts
2026.05.06 08:48:35.719101 [ 32421 ] {} <Test> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): loadDataPart: inserting 8_9_9_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.719172 [ 32421 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Finished loading Active part 8_9_9_0 on disk default
2026.05.06 08:48:35.721144 [ 32421 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loading Active part 3_4_4_0 from disk default
2026.05.06 08:48:35.719393 [ 32427 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_19_19_0 from disk default
2026.05.06 08:48:35.729438 [ 32343 ] {} <Test> test_1.test1 (5d763aa7-9b30-4a61-8f15-11f7ca5b4d88): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.729546 [ 32343 ] {} <Trace> test_1.test1 (5d763aa7-9b30-4a61-8f15-11f7ca5b4d88): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.729988 [ 32339 ] {} <Test> test_13.test_table_2 (97e92dfd-a651-4d7a-94bb-959bc6bd2a6f): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.730051 [ 32339 ] {} <Trace> test_13.test_table_2 (97e92dfd-a651-4d7a-94bb-959bc6bd2a6f): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.730025 [ 32441 ] {} <Trace> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): Loading Active part all_3_3_0 from disk default
2026.05.06 08:48:35.732100 [ 32421 ] {} <Test> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): loadDataPart: inserting 3_4_4_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.732168 [ 32421 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Finished loading Active part 3_4_4_0 on disk default
2026.05.06 08:48:35.732501 [ 32421 ] {} <Trace> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.732511 [ 32397 ] {} <Test> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.732569 [ 32397 ] {} <Trace> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Will load 3 parts using up to 64 threads
2026.05.06 08:48:35.734282 [ 32421 ] {} <Test> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.734357 [ 32421 ] {} <Trace> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.735588 [ 32394 ] {} <Test> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.735653 [ 32394 ] {} <Trace> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.744185 [ 32394 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loading Active part 7_8_8_0 from disk default
2026.05.06 08:48:35.744516 [ 32421 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loading Active part 4_5_5_0 from disk default
2026.05.06 08:48:35.748842 [ 32340 ] {} <Test> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.748903 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_1_130_13 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.748927 [ 32340 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Will load 2 parts using up to 64 threads
2026.05.06 08:48:35.748961 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_131_221_14 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.748990 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_131_252_15 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749018 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_222_232_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749046 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_222_237_3 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749080 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_222_242_4 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749120 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_222_247_5 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749156 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_222_252_6 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749198 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_233_233_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749242 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_234_234_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749272 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_235_235_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749299 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_236_236_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749327 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_237_237_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.747398 [ 32429 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loading Active part 0_1_1_0 from disk default
2026.05.06 08:48:35.748162 [ 32430 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loading Active part 1_2_2_0 from disk default
2026.05.06 08:48:35.749356 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_238_238_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749428 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_239_239_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749457 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_240_240_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749497 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_241_241_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749534 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_242_242_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749561 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_243_243_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749588 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_244_244_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749615 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_245_245_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749642 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_246_246_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749669 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_247_247_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749696 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_248_248_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749722 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_249_249_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749750 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_250_250_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749776 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_251_251_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749803 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_252_252_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749830 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_253_253_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749856 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_253_258_1 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749884 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_253_263_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749910 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_253_268_3 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749937 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_253_273_4 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749963 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_253_278_5 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.749990 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_253_283_6 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.750017 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_253_288_7 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.750043 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_253_294_8 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.750069 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_253_300_9 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.750096 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_254_254_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.750126 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_255_255_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.750155 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_256_256_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.750200 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_257_257_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.750239 [ 32316 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): removePartsFinally: removing 202605_260_260_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:35.750235 [ 32344 ] {} <Test> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): loadDataPart: inserting all_9_9_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.750323 [ 32344 ] {} <Trace> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): Finished loading Active part all_9_9_0 on disk default
2026.05.06 08:48:35.750358 [ 32316 ] {} <Debug> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Removing 43 parts from memory: Parts: [202605_1_130_13, 202605_131_221_14, 202605_131_252_15, 202605_222_232_2, 202605_222_237_3, 202605_222_242_4, 202605_222_247_5, 202605_222_252_6, 202605_233_233_0, 202605_234_234_0, 202605_235_235_0, 202605_236_236_0, 202605_237_237_0, 202605_238_238_0, 202605_239_239_0, 202605_240_240_0, 202605_241_241_0, 202605_242_242_0, 202605_243_243_0, 202605_244_244_0, 202605_245_245_0, 202605_246_246_0, 202605_247_247_0, 202605_248_248_0, 202605_249_249_0, 202605_250_250_0, 202605_251_251_0, 202605_252_252_0, 202605_253_253_0, 202605_253_258_1, 202605_253_263_2, 202605_253_268_3, 202605_253_273_4, 202605_253_278_5, 202605_253_283_6, 202605_253_288_7, 202605_253_294_8, 202605_253_300_9, 202605_254_254_0, 202605_255_255_0, 202605_256_256_0, 202605_257_257_0, 202605_260_260_0]
2026.05.06 08:48:35.750425 [ 32365 ] {} <Test> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): loadDataPart: inserting all_8_8_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.750488 [ 32365 ] {} <Trace> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): Finished loading Active part all_8_8_0 on disk default
2026.05.06 08:48:35.750608 [ 32394 ] {} <Test> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): loadDataPart: inserting 7_8_8_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.750657 [ 32378 ] {} <Test> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): loadDataPart: inserting all_1_6_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.750672 [ 32394 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Finished loading Active part 7_8_8_0 on disk default
2026.05.06 08:48:35.750730 [ 32378 ] {} <Trace> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): Finished loading Active part all_1_6_1 on disk default
2026.05.06 08:48:35.750776 [ 32441 ] {} <Test> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): loadDataPart: inserting all_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.750809 [ 32344 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_7_7_0 from disk default
2026.05.06 08:48:35.750827 [ 32441 ] {} <Trace> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): Finished loading Active part all_3_3_0 on disk default
2026.05.06 08:48:35.750839 [ 32365 ] {} <Trace> test_13.test_table_2 (97e92dfd-a651-4d7a-94bb-959bc6bd2a6f): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.751113 [ 32421 ] {} <Test> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): loadDataPart: inserting 4_5_5_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.751169 [ 32421 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Finished loading Active part 4_5_5_0 on disk default
2026.05.06 08:48:35.751886 [ 32422 ] {} <Test> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): loadDataPart: inserting all_7_7_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.751933 [ 32422 ] {} <Trace> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): Finished loading Active part all_7_7_0 on disk default
2026.05.06 08:48:35.753013 [ 32403 ] {} <Debug> test_13.table_for_freeze_old_syntax (c8eb0055-c2cb-48ce-90ae-0c88b0eb2b2d): Loading data parts
2026.05.06 08:48:35.754814 [ 32394 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loading Active part 6_7_7_0 from disk default
2026.05.06 08:48:35.750323 [ 32436 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loading Active part 9_10_10_0 from disk default
2026.05.06 08:48:35.755266 [ 32378 ] {} <Trace> test_1.test1 (5d763aa7-9b30-4a61-8f15-11f7ca5b4d88): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.755276 [ 32381 ] {} <Debug> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): Loaded data parts (3 items)
2026.05.06 08:48:35.755408 [ 32421 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loading Active part 2_3_3_0 from disk default
2026.05.06 08:48:35.755866 [ 32381 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.755728 [ 32438 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_12_12_0 from disk default
2026.05.06 08:48:35.756248 [ 32441 ] {} <Trace> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Loading Active part all_1_1_0_6 from disk default
2026.05.06 08:48:35.752102 [ 32437 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loading Active part 5_6_6_0 from disk default
2026.05.06 08:48:35.754808 [ 32435 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_16_16_0 from disk default
2026.05.06 08:48:35.757144 [ 32368 ] {} <Debug> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Loading data parts
2026.05.06 08:48:35.757247 [ 32381 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.757320 [ 32422 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Loading Active part all_1_5_1 from disk default
2026.05.06 08:48:35.757328 [ 32381 ] {} <Test> DatabaseAtomic (test_13): Counting attached table array_jaccard_index to database test_13
2026.05.06 08:48:35.757515 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.array_jaccard_index' with status OK
2026.05.06 08:48:35.759218 [ 32344 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_7_7_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.759270 [ 32344 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_7_7_0 on disk default
2026.05.06 08:48:35.759781 [ 32365 ] {} <Test> test_13.test_table_2 (97e92dfd-a651-4d7a-94bb-959bc6bd2a6f): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.759848 [ 32365 ] {} <Trace> test_13.test_table_2 (97e92dfd-a651-4d7a-94bb-959bc6bd2a6f): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.760268 [ 32394 ] {} <Test> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): loadDataPart: inserting 6_7_7_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.760320 [ 32394 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Finished loading Active part 6_7_7_0 on disk default
2026.05.06 08:48:35.763799 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.mt' in BackgroundLoad
2026.05.06 08:48:35.764578 [ 32428 ] {} <Test> test_13.json_test (9048d3f2-57a4-4fab-9e7f-11419ec07f25): loadDataPart: inserting all_1_1_0_2 (state Active) into data_parts_indexes
2026.05.06 08:48:35.764630 [ 32428 ] {} <Trace> test_13.json_test (9048d3f2-57a4-4fab-9e7f-11419ec07f25): Finished loading Active part all_1_1_0_2 on disk default
2026.05.06 08:48:35.764641 [ 32381 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.mt
2026.05.06 08:48:35.764681 [ 32344 ] {} <Trace> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Loading Active part all_3_3_0_6 from disk default
2026.05.06 08:48:35.764750 [ 32428 ] {} <Trace> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Loading Active part all_5_5_0_6 from disk default
2026.05.06 08:48:35.764745 [ 32370 ] {} <Debug> test_13.json_test (9048d3f2-57a4-4fab-9e7f-11419ec07f25): Loaded data parts (1 items)
2026.05.06 08:48:35.764783 [ 32339 ] {} <Debug> test_13.test_table_2 (97e92dfd-a651-4d7a-94bb-959bc6bd2a6f): Loaded data parts (1 items)
2026.05.06 08:48:35.764804 [ 32365 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_6_6_0 from disk default
2026.05.06 08:48:35.764690 [ 32394 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Loading Active part all_6_6_0 from disk default
2026.05.06 08:48:35.765565 [ 32422 ] {} <Test> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): loadDataPart: inserting all_1_5_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.765609 [ 32422 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Finished loading Active part all_1_5_1 on disk default
2026.05.06 08:48:35.766888 [ 32421 ] {} <Test> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): loadDataPart: inserting 2_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.766935 [ 32421 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Finished loading Active part 2_3_3_0 on disk default
2026.05.06 08:48:35.768197 [ 32422 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_3_3_0 from disk default
2026.05.06 08:48:35.768233 [ 32339 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.768236 [ 32421 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_8_8_0 from disk default
2026.05.06 08:48:35.768297 [ 32339 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.769323 [ 32344 ] {} <Test> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): loadDataPart: inserting all_3_3_0_6 (state Active) into data_parts_indexes
2026.05.06 08:48:35.769367 [ 32344 ] {} <Trace> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Finished loading Active part all_3_3_0_6 on disk default
2026.05.06 08:48:35.769809 [ 32394 ] {} <Test> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): loadDataPart: inserting all_6_6_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.769860 [ 32394 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Finished loading Active part all_6_6_0 on disk default
2026.05.06 08:48:35.770430 [ 32339 ] {} <Test> DatabaseAtomic (test_13): Counting attached table test_table_2 to database test_13
2026.05.06 08:48:35.770584 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.test_table_2' with status OK
2026.05.06 08:48:35.771145 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.t1' in BackgroundLoad
2026.05.06 08:48:35.771205 [ 32339 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.t1
2026.05.06 08:48:35.771324 [ 32365 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_6_6_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.771369 [ 32365 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_6_6_0 on disk default
2026.05.06 08:48:35.771416 [ 32340 ] {} <Debug> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Found 5 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:35.771498 [ 32340 ] {} <Debug> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Loaded data parts (2 items)
2026.05.06 08:48:35.771529 [ 32344 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_17_17_0 from disk default
2026.05.06 08:48:35.771545 [ 32394 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_18_18_0 from disk default
2026.05.06 08:48:35.771612 [ 32365 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_4_4_0 from disk default
2026.05.06 08:48:35.771646 [ 32422 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.771695 [ 32422 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_3_3_0 on disk default
2026.05.06 08:48:35.771917 [ 32340 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.772233 [ 32428 ] {} <Test> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): loadDataPart: inserting all_5_5_0_6 (state Active) into data_parts_indexes
2026.05.06 08:48:35.772284 [ 32428 ] {} <Trace> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Finished loading Active part all_5_5_0_6 on disk default
2026.05.06 08:48:35.774052 [ 32340 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.774120 [ 32340 ] {} <Test> DatabaseAtomic (test_1): Counting attached table test_collation to database test_1
2026.05.06 08:48:35.774281 [ 32340 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.test_collation' with status OK
2026.05.06 08:48:35.774351 [ 32340 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.rmt1' in BackgroundLoad
2026.05.06 08:48:35.774390 [ 32340 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.rmt1
2026.05.06 08:48:35.774893 [ 32421 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_8_8_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.774940 [ 32421 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_8_8_0 on disk default
2026.05.06 08:48:35.776884 [ 32365 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_4_4_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.776929 [ 32365 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_4_4_0 on disk default
2026.05.06 08:48:35.777318 [ 32441 ] {} <Test> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): loadDataPart: inserting all_1_1_0_6 (state Active) into data_parts_indexes
2026.05.06 08:48:35.777367 [ 32441 ] {} <Trace> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Finished loading Active part all_1_1_0_6 on disk default
2026.05.06 08:48:35.778434 [ 32430 ] {} <Test> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): loadDataPart: inserting 1_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.778509 [ 32430 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Finished loading Active part 1_2_2_0 on disk default
2026.05.06 08:48:35.779012 [ 32397 ] {} <Debug> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Loaded data parts (3 items)
2026.05.06 08:48:35.779182 [ 32427 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_19_19_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.779246 [ 32427 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_19_19_0 on disk default
2026.05.06 08:48:35.780503 [ 32401 ] {} <Debug> test_13.t_index_hypothesis (ab916a7e-4ba1-46ee-91e7-5ef0bd94aa88): Loaded data parts (4 items)
2026.05.06 08:48:35.780891 [ 32394 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_18_18_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.780949 [ 32394 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_18_18_0 on disk default
2026.05.06 08:48:35.785355 [ 32339 ] {} <Debug> test_15.t1 (eebf517e-6b98-4afe-aed1-789e6d84f79d): Loading data parts
2026.05.06 08:48:35.785729 [ 32403 ] {} <Test> test_13.table_for_freeze_old_syntax (c8eb0055-c2cb-48ce-90ae-0c88b0eb2b2d): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.785812 [ 32403 ] {} <Trace> test_13.table_for_freeze_old_syntax (c8eb0055-c2cb-48ce-90ae-0c88b0eb2b2d): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.792288 [ 32339 ] {} <Test> test_15.t1 (eebf517e-6b98-4afe-aed1-789e6d84f79d): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.792346 [ 32339 ] {} <Debug> test_15.t1 (eebf517e-6b98-4afe-aed1-789e6d84f79d): There are no data parts
2026.05.06 08:48:35.792816 [ 32339 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.792186 [ 32440 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_11_11_0 from disk default
2026.05.06 08:48:35.794020 [ 32442 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_9_9_0 from disk default
2026.05.06 08:48:35.791922 [ 32449 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_10_10_0 from disk default
2026.05.06 08:48:35.791893 [ 32446 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_5_5_0 from disk default
2026.05.06 08:48:35.794150 [ 32397 ] {} <Debug> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Loading mutation: mutation_2.txt entry, commands size: 1
2026.05.06 08:48:35.791890 [ 32434 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_15_15_0 from disk default
2026.05.06 08:48:35.794238 [ 32368 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.794306 [ 32368 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Will load 3 parts using up to 64 threads
2026.05.06 08:48:35.794611 [ 32339 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.794680 [ 32339 ] {} <Test> DatabaseAtomic (test_15): Counting attached table t1 to database test_15
2026.05.06 08:48:35.794758 [ 32429 ] {} <Test> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): loadDataPart: inserting 0_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.794813 [ 32429 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Finished loading Active part 0_1_1_0 on disk default
2026.05.06 08:48:35.794829 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.t1' with status OK
2026.05.06 08:48:35.794908 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.defaulted' in BackgroundLoad
2026.05.06 08:48:35.794959 [ 32339 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.defaulted
2026.05.06 08:48:35.795009 [ 32344 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_17_17_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.795061 [ 32344 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_17_17_0 on disk default
2026.05.06 08:48:35.792182 [ 32444 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_13_13_0 from disk default
2026.05.06 08:48:35.799921 [ 32436 ] {} <Test> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): loadDataPart: inserting 9_10_10_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.799975 [ 32436 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Finished loading Active part 9_10_10_0 on disk default
2026.05.06 08:48:35.800014 [ 32365 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Loading Active part 20191003_0_0_0_1 from disk default
2026.05.06 08:48:35.800022 [ 32421 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Loading Active part 20191001_0_0_0_1 from disk default
2026.05.06 08:48:35.800216 [ 32401 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.800280 [ 32441 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Loading Active part 20191002_0_0_0_1 from disk default
2026.05.06 08:48:35.800764 [ 32401 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.803386 [ 32421 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): loadDataPart: inserting 20191001_0_0_0_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.803447 [ 32421 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Finished loading Active part 20191001_0_0_0_1 on disk default
2026.05.06 08:48:35.805198 [ 32438 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_12_12_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.805273 [ 32438 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_12_12_0 on disk default
2026.05.06 08:48:35.811996 [ 32401 ] {} <Test> DatabaseAtomic (test_13): Counting attached table t_index_hypothesis to database test_13
2026.05.06 08:48:35.813475 [ 32441 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): loadDataPart: inserting 20191002_0_0_0_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.813556 [ 32441 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Finished loading Active part 20191002_0_0_0_1 on disk default
2026.05.06 08:48:35.817572 [ 32401 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.t_index_hypothesis' with status OK
2026.05.06 08:48:35.817669 [ 32401 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.my_events' in BackgroundLoad
2026.05.06 08:48:35.817726 [ 32401 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.my_events
2026.05.06 08:48:35.818036 [ 32437 ] {} <Test> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): loadDataPart: inserting 5_6_6_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.818110 [ 32437 ] {} <Trace> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Finished loading Active part 5_6_6_0 on disk default
2026.05.06 08:48:35.818292 [ 32371 ] {} <Debug> test_truncate_database.source_table_merge_tree (7c413161-9dd2-4883-8840-e0021074db90): Loaded data parts (10 items)
2026.05.06 08:48:35.818566 [ 32365 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): loadDataPart: inserting 20191003_0_0_0_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.818624 [ 32365 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Finished loading Active part 20191003_0_0_0_1 on disk default
2026.05.06 08:48:35.818702 [ 32435 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_16_16_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.818752 [ 32435 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_16_16_0 on disk default
2026.05.06 08:48:35.818817 [ 32368 ] {} <Debug> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Loaded data parts (3 items)
2026.05.06 08:48:35.818919 [ 32368 ] {} <Information> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.818990 [ 32371 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.819049 [ 32371 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.819106 [ 32371 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table source_table_merge_tree to database test_truncate_database
2026.05.06 08:48:35.819216 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.source_table_merge_tree' with status OK
2026.05.06 08:48:35.819270 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.dim_model' in BackgroundLoad
2026.05.06 08:48:35.819306 [ 32371 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.dim_model
2026.05.06 08:48:35.819589 [ 32371 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.819637 [ 32371 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.819679 [ 32449 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_10_10_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.819715 [ 32371 ] {} <Test> DatabaseAtomic (test_1): Counting attached table dim_model to database test_1
2026.05.06 08:48:35.819736 [ 32449 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_10_10_0 on disk default
2026.05.06 08:48:35.819817 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.dim_model' with status OK
2026.05.06 08:48:35.819878 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.t1__fuzz_26' in BackgroundLoad
2026.05.06 08:48:35.819918 [ 32371 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.t1__fuzz_26
2026.05.06 08:48:35.819943 [ 32370 ] {} <Debug> test_13.json_test (9048d3f2-57a4-4fab-9e7f-11419ec07f25): Loading mutation: mutation_2.txt entry, commands size: 1
2026.05.06 08:48:35.820027 [ 32340 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/lbj/yuuylwgvmlxkpfzqqzcdlcetbhqni (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/format_version.txt)
2026.05.06 08:48:35.820317 [ 32397 ] {} <Debug> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Loading mutation: mutation_6.txt entry, commands size: 2
2026.05.06 08:48:35.820332 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(f069604742ee525a04e9e016584d3212): Reset state for seek to position 0
2026.05.06 08:48:35.820611 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(f069604742ee525a04e9e016584d3212): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:35.820657 [ 32397 ] {} <Debug> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Loading mutation: mutation_4.txt entry, commands size: 1
2026.05.06 08:48:35.820908 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(f069604742ee525a04e9e016584d3212): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: f069604742ee525a04e9e016584d3212, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:BackgroundLoad:32340, kind: Regular, unbound: 0
2026.05.06 08:48:35.821049 [ 32340 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -1, current is 2772
2026.05.06 08:48:35.821140 [ 32376 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataPart: inserting 5_13_13_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.821197 [ 32376 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Finished loading Active part 5_13_13_0 on disk default
2026.05.06 08:48:35.821207 [ 32340 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: f069604742ee525a04e9e016584d3212, offset: 0, size: 1
2026.05.06 08:48:35.821337 [ 32340 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 1, current is 0
2026.05.06 08:48:35.821457 [ 32340 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: f069604742ee525a04e9e016584d3212, offset: 0, size: 1
2026.05.06 08:48:35.821772 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(f069604742ee525a04e9e016584d3212): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: f069604742ee525a04e9e016584d3212, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:BackgroundLoad:32340, kind: Regular, unbound: 0
2026.05.06 08:48:35.821212 [ 32387 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataPart: inserting 4_5_5_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.821893 [ 32423 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataPart: inserting 8_9_9_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.821892 [ 32387 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Finished loading Active part 4_5_5_0 on disk default
2026.05.06 08:48:35.821944 [ 32442 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_9_9_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.821952 [ 32423 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Finished loading Active part 8_9_9_0 on disk default
2026.05.06 08:48:35.821985 [ 32424 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataPart: inserting 1_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.822005 [ 32442 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_9_9_0 on disk default
2026.05.06 08:48:35.822012 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(f069604742ee525a04e9e016584d3212): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:35.822061 [ 32426 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataPart: inserting 6_7_7_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.822046 [ 32424 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Finished loading Active part 1_2_2_0 on disk default
2026.05.06 08:48:35.822140 [ 32426 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Finished loading Active part 6_7_7_0 on disk default
2026.05.06 08:48:35.822174 [ 32342 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataPart: inserting 3_12_12_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.822238 [ 32342 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Finished loading Active part 3_12_12_0 on disk default
2026.05.06 08:48:35.822253 [ 32418 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataPart: inserting 9_10_10_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.822332 [ 32418 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Finished loading Active part 9_10_10_0 on disk default
2026.05.06 08:48:35.822333 [ 32425 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataPart: inserting 3_11_11_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.822397 [ 32425 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Finished loading Active part 3_11_11_0 on disk default
2026.05.06 08:48:35.822456 [ 32388 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataPart: inserting 7_8_8_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.822565 [ 32388 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Finished loading Active part 7_8_8_0 on disk default
2026.05.06 08:48:35.822772 [ 32351 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataPart: inserting 0_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.822557 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(f069604742ee525a04e9e016584d3212): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:35.822841 [ 32351 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Finished loading Active part 0_1_1_0 on disk default
2026.05.06 08:48:35.823084 [ 32337 ] {} <Test> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): loadDataPart: inserting 2_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.823135 [ 32337 ] {} <Trace> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Finished loading Active part 2_3_3_0 on disk default
2026.05.06 08:48:35.823296 [ 32402 ] {} <Debug> test_13.table_for_freeze (763743a3-b661-4442-be93-0d7aee8e21fa): Loaded data parts (11 items)
2026.05.06 08:48:35.823173 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(f069604742ee525a04e9e016584d3212): Key: f069604742ee525a04e9e016584d3212. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:35.818990 [ 32339 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.824369 [ 32339 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.824449 [ 32339 ] {} <Test> DatabaseAtomic (test_13): Counting attached table defaulted to database test_13
2026.05.06 08:48:35.824531 [ 32378 ] {} <Test> test_1.test1 (5d763aa7-9b30-4a61-8f15-11f7ca5b4d88): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.824535 [ 32368 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.824629 [ 32368 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.824699 [ 32446 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_5_5_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.824715 [ 32368 ] {} <Test> DatabaseAtomic (test_13): Counting attached table table_for_rename_replicated to database test_13
2026.05.06 08:48:35.824772 [ 32446 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_5_5_0 on disk default
2026.05.06 08:48:35.824813 [ 32378 ] {} <Trace> test_1.test1 (5d763aa7-9b30-4a61-8f15-11f7ca5b4d88): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.824815 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.defaulted' with status OK
2026.05.06 08:48:35.824806 [ 32434 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_15_15_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.824858 [ 32370 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.824937 [ 32434 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_15_15_0 on disk default
2026.05.06 08:48:35.824946 [ 32343 ] {} <Debug> test_1.test1 (5d763aa7-9b30-4a61-8f15-11f7ca5b4d88): Loaded data parts (1 items)
2026.05.06 08:48:35.824966 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.t2' in BackgroundLoad
2026.05.06 08:48:35.824941 [ 32370 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.825122 [ 32339 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.t2
2026.05.06 08:48:35.825140 [ 32370 ] {} <Test> DatabaseAtomic (test_13): Counting attached table json_test to database test_13
2026.05.06 08:48:35.825216 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.825244 [ 32440 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_11_11_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.825262 [ 32368 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.table_for_rename_replicated' with status OK
2026.05.06 08:48:35.825271 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.825298 [ 32440 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_11_11_0 on disk default
2026.05.06 08:48:35.825338 [ 32368 ] {} <Debug> AsyncLoader: Execute load job 'load table test_m91yt8r2_1.t' in BackgroundLoad
2026.05.06 08:48:35.825347 [ 32397 ] {} <Test> DatabaseAtomic (test_13): Counting attached table rename_table_multiple to database test_13
2026.05.06 08:48:35.825416 [ 32368 ] {} <Trace> DatabaseAtomic (test_m91yt8r2_1): Loading table test_m91yt8r2_1.t
2026.05.06 08:48:35.825468 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.825475 [ 32402 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.825534 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.json_test' with status OK
2026.05.06 08:48:35.825566 [ 32381 ] {} <Debug> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): Loading data parts
2026.05.06 08:48:35.825569 [ 32402 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.825569 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.825634 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.table_for_freeze_replicated' in BackgroundLoad
2026.05.06 08:48:35.825676 [ 32402 ] {} <Test> DatabaseAtomic (test_13): Counting attached table table_for_freeze to database test_13
2026.05.06 08:48:35.825687 [ 32343 ] {} <Test> DatabaseAtomic (test_1): Counting attached table test1 to database test_1
2026.05.06 08:48:35.825708 [ 32370 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.table_for_freeze_replicated
2026.05.06 08:48:35.825796 [ 32343 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.test1' with status OK
2026.05.06 08:48:35.825853 [ 32343 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.t' in BackgroundLoad
2026.05.06 08:48:35.825931 [ 32444 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_13_13_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.825936 [ 32343 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.t
2026.05.06 08:48:35.825953 [ 32368 ] {} <Debug> DatabaseAtomic (test_m91yt8r2_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.825971 [ 32377 ] {} <Test> test_13.test (40407d88-838f-4821-9923-ec6b735cb457): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.826000 [ 32444 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_13_13_0 on disk default
2026.05.06 08:48:35.826033 [ 32377 ] {} <Trace> test_13.test (40407d88-838f-4821-9923-ec6b735cb457): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.826064 [ 32402 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.table_for_freeze' with status OK
2026.05.06 08:48:35.826104 [ 32368 ] {} <Debug> DatabaseAtomic (test_m91yt8r2_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.826132 [ 32402 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.mv_b_even' in BackgroundLoad
2026.05.06 08:48:35.826141 [ 32419 ] {} <Test> test_15.t_proj_external (87447c1e-c29e-4e0e-af58-72cfc77edc4b): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.826154 [ 32390 ] {} <Debug> test_13.test (40407d88-838f-4821-9923-ec6b735cb457): Loaded data parts (1 items)
2026.05.06 08:48:35.826178 [ 32402 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.mv_b_even
2026.05.06 08:48:35.826167 [ 32368 ] {} <Test> DatabaseAtomic (test_m91yt8r2_1): Counting attached table t to database test_m91yt8r2_1
2026.05.06 08:48:35.826220 [ 32371 ] {} <Debug> test_15.t1__fuzz_26 (d2f1f23e-b6bc-4a1e-bad9-26c65a299f6c): Loading data parts
2026.05.06 08:48:35.826196 [ 32419 ] {} <Trace> test_15.t_proj_external (87447c1e-c29e-4e0e-af58-72cfc77edc4b): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.826282 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.rename_table_multiple' with status OK
2026.05.06 08:48:35.826319 [ 32401 ] {} <Debug> test_13.my_events (103ec94e-af48-4926-a1b4-04416daf6ac1): Loading data parts
2026.05.06 08:48:35.826338 [ 32381 ] {} <Test> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.826462 [ 32340 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Loading data parts
2026.05.06 08:48:35.827443 [ 32381 ] {} <Trace> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): Will load 3 parts using up to 64 threads
2026.05.06 08:48:35.827774 [ 32371 ] {} <Test> test_15.t1__fuzz_26 (d2f1f23e-b6bc-4a1e-bad9-26c65a299f6c): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.827786 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.h3_indexes' in BackgroundLoad
2026.05.06 08:48:35.827815 [ 32398 ] {} <Debug> test_15.t_proj_external (87447c1e-c29e-4e0e-af58-72cfc77edc4b): Loaded data parts (2 items)
2026.05.06 08:48:35.827844 [ 32371 ] {} <Debug> test_15.t1__fuzz_26 (d2f1f23e-b6bc-4a1e-bad9-26c65a299f6c): There are no data parts
2026.05.06 08:48:35.827864 [ 32397 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.h3_indexes
2026.05.06 08:48:35.827965 [ 32368 ] {} <Debug> AsyncLoader: Finish load job 'load table test_m91yt8r2_1.t' with status OK
2026.05.06 08:48:35.827428 [ 32431 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_20_20_0 from disk default
2026.05.06 08:48:35.827381 [ 32439 ] {} <Trace> test_13.table_for_freeze_old_syntax (c8eb0055-c2cb-48ce-90ae-0c88b0eb2b2d): Loading Active part 20210301_20210301_1_1_0 from disk default
2026.05.06 08:48:35.828074 [ 32401 ] {} <Test> test_13.my_events (103ec94e-af48-4926-a1b4-04416daf6ac1): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.828141 [ 32401 ] {} <Trace> test_13.my_events (103ec94e-af48-4926-a1b4-04416daf6ac1): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.828169 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.828191 [ 32402 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.827474 [ 32443 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_14_14_0 from disk default
2026.05.06 08:48:35.827653 [ 32448 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.827353 [ 32445 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.828292 [ 32371 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.828356 [ 31633 ] {} <Trace> AsynchronousMetrics: Update heavy metrics. Update period 1 sec. Update heavy metrics period 120 sec. Heavy metrics calculation elapsed: 0.032792801 sec.
2026.05.06 08:48:35.828452 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk 03008_s3_plain_rewritable eligible for search: false (mode ANY)
2026.05.06 08:48:35.828641 [ 32368 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.data_01279' in BackgroundLoad
2026.05.06 08:48:35.828699 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.828701 [ 32368 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.data_01279
2026.05.06 08:48:35.828722 [ 32402 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.828757 [ 32397 ] {} <Test> DatabaseAtomic (test_13): Counting attached table h3_indexes to database test_13
2026.05.06 08:48:35.828811 [ 32390 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.828825 [ 32451 ] {} <Trace> test_13.my_events (103ec94e-af48-4926-a1b4-04416daf6ac1): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.828867 [ 32371 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.828869 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.h3_indexes' with status OK
2026.05.06 08:48:35.828945 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.buf' in BackgroundLoad
2026.05.06 08:48:35.828980 [ 32397 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.buf
2026.05.06 08:48:35.829139 [ 32402 ] {} <Test> DatabaseAtomic (test_1): Counting attached table mv_b_even to database test_1
2026.05.06 08:48:35.829293 [ 32402 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.mv_b_even' with status OK
2026.05.06 08:48:35.829542 [ 32371 ] {} <Test> DatabaseAtomic (test_15): Counting attached table t1__fuzz_26 to database test_15
2026.05.06 08:48:35.829567 [ 32390 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.829602 [ 32402 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.mv_source' in BackgroundLoad
2026.05.06 08:48:35.829600 [ 32398 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.829634 [ 32390 ] {} <Test> DatabaseAtomic (test_13): Counting attached table test to database test_13
2026.05.06 08:48:35.829663 [ 32402 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.mv_source
2026.05.06 08:48:35.829667 [ 32398 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.829699 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.829721 [ 32398 ] {} <Test> DatabaseAtomic (test_15): Counting attached table t_proj_external to database test_15
2026.05.06 08:48:35.829740 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.829785 [ 32397 ] {} <Test> DatabaseAtomic (test_13): Counting attached table buf to database test_13
2026.05.06 08:48:35.829786 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.t1__fuzz_26' with status OK
2026.05.06 08:48:35.829783 [ 32368 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.831776 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk azure eligible for search: true (mode ANY)
2026.05.06 08:48:35.832287 [ 32368 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.832302 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.replicated_mutation_table' in BackgroundLoad
2026.05.06 08:48:35.832453 [ 32368 ] {} <Test> DatabaseAtomic (test_15): Counting attached table data_01279 to database test_15
2026.05.06 08:48:35.832496 [ 32371 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.replicated_mutation_table
2026.05.06 08:48:35.832795 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk backups eligible for search: true (mode ANY)
2026.05.06 08:48:35.832880 [ 32390 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.test' with status OK
2026.05.06 08:48:35.832948 [ 32390 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.date_t' in BackgroundLoad
2026.05.06 08:48:35.832963 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk azure eligible for search: true (mode ANY)
2026.05.06 08:48:35.832996 [ 32390 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.date_t
2026.05.06 08:48:35.833049 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk default eligible for search: true (mode ANY)
2026.05.06 08:48:35.833175 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk disk1_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.833181 [ 32398 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.t_proj_external' with status OK
2026.05.06 08:48:35.833261 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk disk2_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.833294 [ 32398 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.collapsing_table' in BackgroundLoad
2026.05.06 08:48:35.833320 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk disk3_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.833353 [ 32398 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.collapsing_table
2026.05.06 08:48:35.833380 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk disk4_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.833438 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.833522 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.833594 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk local_disk_3 eligible for search: true (mode ANY)
2026.05.06 08:48:35.833641 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.833719 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk local_disk_2 eligible for search: true (mode ANY)
2026.05.06 08:48:35.833778 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk local_disk_3 eligible for search: true (mode ANY)
2026.05.06 08:48:35.833836 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.833854 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.buf' with status OK
2026.05.06 08:48:35.833879 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.833920 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.data_01670' in BackgroundLoad
2026.05.06 08:48:35.833925 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.833976 [ 32397 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.data_01670
2026.05.06 08:48:35.834062 [ 32368 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.data_01279' with status OK
2026.05.06 08:48:35.833773 [ 32450 ] {} <Trace> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): Loading Active part 20190301_1_1_0 from disk default
2026.05.06 08:48:35.834581 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.834621 [ 32374 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/dcotppgaoojjrgvmizsltsrzracgrgkj/format_version.txt (store/de4/de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b/format_version.txt)
2026.05.06 08:48:35.834647 [ 32339 ] {} <Debug> test_15.t2 (4c092d00-faaa-4954-86be-533f892efa40): Loading data parts
2026.05.06 08:48:35.834903 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.834885 [ 32433 ] {} <Trace> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): Loading Active part 20190303_3_3_0 from disk default
2026.05.06 08:48:35.834634 [ 32368 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.mydb' in BackgroundLoad
2026.05.06 08:48:35.834969 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.834635 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.835030 [ 32368 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.mydb
2026.05.06 08:48:35.835063 [ 32397 ] {} <Test> DatabaseAtomic (test_13): Counting attached table data_01670 to database test_13
2026.05.06 08:48:35.835179 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.data_01670' with status OK
2026.05.06 08:48:35.835405 [ 32439 ] {} <Test> test_13.table_for_freeze_old_syntax (c8eb0055-c2cb-48ce-90ae-0c88b0eb2b2d): loadDataPart: inserting 20210301_20210301_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.835495 [ 32439 ] {} <Trace> test_13.table_for_freeze_old_syntax (c8eb0055-c2cb-48ce-90ae-0c88b0eb2b2d): Finished loading Active part 20210301_20210301_1_1_0 on disk default
2026.05.06 08:48:35.835731 [ 32451 ] {} <Test> test_13.my_events (103ec94e-af48-4926-a1b4-04416daf6ac1): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.835782 [ 32451 ] {} <Trace> test_13.my_events (103ec94e-af48-4926-a1b4-04416daf6ac1): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.835976 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.836022 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk s3_disk_02944 eligible for search: true (mode ANY)
2026.05.06 08:48:35.836073 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk s3_disk_02963 eligible for search: true (mode ANY)
2026.05.06 08:48:35.836116 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk s3_no_cache eligible for search: true (mode ANY)
2026.05.06 08:48:35.836160 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): is disk s3_plain_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.836265 [ 32402 ] {} <Debug> test_13.mv_source (faa5df6a-eff8-45fb-bbba-2f5d2001af22): Loading data parts
2026.05.06 08:48:35.836480 [ 32368 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.836541 [ 32368 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.836554 [ 32431 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_20_20_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.836600 [ 32368 ] {} <Test> DatabaseAtomic (test_15): Counting attached table mydb to database test_15
2026.05.06 08:48:35.836631 [ 32431 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_20_20_0 on disk default
2026.05.06 08:48:35.836695 [ 32402 ] {} <Test> test_13.mv_source (faa5df6a-eff8-45fb-bbba-2f5d2001af22): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.836773 [ 32402 ] {} <Trace> test_13.mv_source (faa5df6a-eff8-45fb-bbba-2f5d2001af22): Will load 2 parts using up to 64 threads
2026.05.06 08:48:35.836914 [ 32443 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_14_14_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.836952 [ 32344 ] {} <Trace> test_13.mv_source (faa5df6a-eff8-45fb-bbba-2f5d2001af22): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.836969 [ 32436 ] {} <Trace> test_13.mv_source (faa5df6a-eff8-45fb-bbba-2f5d2001af22): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.836981 [ 32443 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_14_14_0 on disk default
2026.05.06 08:48:35.837139 [ 32403 ] {} <Debug> test_13.table_for_freeze_old_syntax (c8eb0055-c2cb-48ce-90ae-0c88b0eb2b2d): Loaded data parts (1 items)
2026.05.06 08:48:35.837134 [ 31633 ] {} <Trace> ExternalDictionariesLoader: Loading config file '/etc/clickhouse-server/decimals_dictionary.xml'.
2026.05.06 08:48:35.837377 [ 32448 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.837423 [ 32448 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.837814 [ 32403 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.837852 [ 32403 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.838153 [ 32403 ] {} <Test> DatabaseAtomic (test_13): Counting attached table table_for_freeze_old_syntax to database test_13
2026.05.06 08:48:35.838589 [ 32344 ] {} <Test> test_13.mv_source (faa5df6a-eff8-45fb-bbba-2f5d2001af22): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.838642 [ 32344 ] {} <Trace> test_13.mv_source (faa5df6a-eff8-45fb-bbba-2f5d2001af22): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.838660 [ 32398 ] {} <Debug> test_13.collapsing_table (94c5b861-aaff-4d33-97f7-126faaa149e6): Loading data parts
2026.05.06 08:48:35.838760 [ 32450 ] {} <Test> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): loadDataPart: inserting 20190301_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.838822 [ 32450 ] {} <Trace> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): Finished loading Active part 20190301_1_1_0 on disk default
2026.05.06 08:48:35.838847 [ 32390 ] {} <Debug> test_13.date_t (90955474-921b-4bbe-837d-ed1227ebb8b1): Loading data parts
2026.05.06 08:48:35.838945 [ 32398 ] {} <Test> test_13.collapsing_table (94c5b861-aaff-4d33-97f7-126faaa149e6): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.838997 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.buffer_00763_1' in BackgroundLoad
2026.05.06 08:48:35.839017 [ 32401 ] {} <Debug> test_13.my_events (103ec94e-af48-4926-a1b4-04416daf6ac1): Loaded data parts (1 items)
2026.05.06 08:48:35.839004 [ 32398 ] {} <Trace> test_13.collapsing_table (94c5b861-aaff-4d33-97f7-126faaa149e6): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.839073 [ 32397 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.buffer_00763_1
2026.05.06 08:48:35.839114 [ 32390 ] {} <Test> test_13.date_t (90955474-921b-4bbe-837d-ed1227ebb8b1): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.839158 [ 32368 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.mydb' with status OK
2026.05.06 08:48:35.839188 [ 32390 ] {} <Debug> test_13.date_t (90955474-921b-4bbe-837d-ed1227ebb8b1): There are no data parts
2026.05.06 08:48:35.839219 [ 32436 ] {} <Test> test_13.mv_source (faa5df6a-eff8-45fb-bbba-2f5d2001af22): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.839298 [ 32436 ] {} <Trace> test_13.mv_source (faa5df6a-eff8-45fb-bbba-2f5d2001af22): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.839412 [ 32438 ] {} <Trace> test_13.collapsing_table (94c5b861-aaff-4d33-97f7-126faaa149e6): Loading Active part all_1_1_1 from disk default
2026.05.06 08:48:35.839448 [ 32401 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.839447 [ 32339 ] {} <Test> test_15.t2 (4c092d00-faaa-4954-86be-533f892efa40): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.839519 [ 32339 ] {} <Trace> test_15.t2 (4c092d00-faaa-4954-86be-533f892efa40): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.839618 [ 32368 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.dict_02916_dictionary_access_test_13' in BackgroundLoad
2026.05.06 08:48:35.839651 [ 32401 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.839649 [ 32437 ] {} <Trace> test_15.t2 (4c092d00-faaa-4954-86be-533f892efa40): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.839681 [ 32433 ] {} <Test> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): loadDataPart: inserting 20190303_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.839689 [ 32402 ] {} <Debug> test_13.mv_source (faa5df6a-eff8-45fb-bbba-2f5d2001af22): Loaded data parts (2 items)
2026.05.06 08:48:35.839715 [ 32401 ] {} <Test> DatabaseAtomic (test_13): Counting attached table my_events to database test_13
2026.05.06 08:48:35.839693 [ 32368 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.dict_02916_dictionary_access_test_13
2026.05.06 08:48:35.839764 [ 32374 ] {} <Debug> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): Loading data parts
2026.05.06 08:48:35.839842 [ 32403 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.table_for_freeze_old_syntax' with status OK
2026.05.06 08:48:35.839902 [ 32397 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.839733 [ 32433 ] {} <Trace> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): Finished loading Active part 20190303_3_3_0 on disk default
2026.05.06 08:48:35.839553 [ 32432 ] {} <Trace> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): Loading Active part 20190302_2_2_0 from disk default
2026.05.06 08:48:35.839991 [ 32397 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.839912 [ 32403 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.table_with_column_ttl' in BackgroundLoad
2026.05.06 08:48:35.840039 [ 32390 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.840070 [ 32397 ] {} <Test> DatabaseAtomic (test_15): Counting attached table buffer_00763_1 to database test_15
2026.05.06 08:48:35.840081 [ 32403 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.table_with_column_ttl
2026.05.06 08:48:35.840160 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk 03008_s3_plain_rewritable eligible for search: false (mode ANY)
2026.05.06 08:48:35.840171 [ 32445 ] {} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.840224 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.buffer_00763_1' with status OK
2026.05.06 08:48:35.840286 [ 32445 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.840321 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.source_table_dictionary' in BackgroundLoad
2026.05.06 08:48:35.840346 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk azure eligible for search: true (mode ANY)
2026.05.06 08:48:35.840391 [ 32397 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.source_table_dictionary
2026.05.06 08:48:35.840440 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk backups eligible for search: true (mode ANY)
2026.05.06 08:48:35.840469 [ 32401 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.my_events' with status OK
2026.05.06 08:48:35.840520 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk azure eligible for search: true (mode ANY)
2026.05.06 08:48:35.840534 [ 32401 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.x' in BackgroundLoad
2026.05.06 08:48:35.840515 [ 32395 ] {} <Debug> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Loaded data parts (20 items)
2026.05.06 08:48:35.840626 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk default eligible for search: true (mode ANY)
2026.05.06 08:48:35.840646 [ 32401 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.x
2026.05.06 08:48:35.840673 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk disk1_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.840716 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk disk2_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.840756 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk disk3_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.840800 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk disk4_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.840808 [ 32397 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.840791 [ 32353 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/lwgrbndnlfdvqzeywyvkzcmdowubxvjw/format_version.txt (store/fef/fef9e575-f89d-43f4-933c-47fb46bf5de1/format_version.txt)
2026.05.06 08:48:35.840847 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.840864 [ 32397 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.840959 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.840996 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk local_disk_3 eligible for search: true (mode ANY)
2026.05.06 08:48:35.840995 [ 32397 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table source_table_dictionary to database test_truncate_database
2026.05.06 08:48:35.841027 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.841071 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk local_disk_2 eligible for search: true (mode ANY)
2026.05.06 08:48:35.841110 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.source_table_dictionary' with status OK
2026.05.06 08:48:35.841116 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk local_disk_3 eligible for search: true (mode ANY)
2026.05.06 08:48:35.841189 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.841189 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.input_function_table_2' in BackgroundLoad
2026.05.06 08:48:35.841222 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.841235 [ 32397 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.input_function_table_2
2026.05.06 08:48:35.841292 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.841317 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.841321 [ 32437 ] {} <Test> test_15.t2 (4c092d00-faaa-4954-86be-533f892efa40): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.841352 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.841350 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.841364 [ 32437 ] {} <Trace> test_15.t2 (4c092d00-faaa-4954-86be-533f892efa40): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.841394 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.841443 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.841443 [ 32395 ] {} <Test> DatabaseAtomic (test_1): Counting attached table table_when_b_even to database test_1
2026.05.06 08:48:35.841491 [ 32339 ] {} <Debug> test_15.t2 (4c092d00-faaa-4954-86be-533f892efa40): Loaded data parts (1 items)
2026.05.06 08:48:35.841510 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk s3_disk_02944 eligible for search: true (mode ANY)
2026.05.06 08:48:35.841621 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk s3_disk_02963 eligible for search: true (mode ANY)
2026.05.06 08:48:35.841627 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.table_when_b_even' with status OK
2026.05.06 08:48:35.841664 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk s3_no_cache eligible for search: true (mode ANY)
2026.05.06 08:48:35.841713 [ 32374 ] {} <Trace> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): is disk s3_plain_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.841729 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table 01778_db.hierarchy_source_table' in BackgroundLoad
2026.05.06 08:48:35.841733 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.841791 [ 32395 ] {} <Trace> DatabaseAtomic (01778_db): Loading table 01778_db.hierarchy_source_table
2026.05.06 08:48:35.841797 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.841852 [ 32397 ] {} <Test> DatabaseAtomic (test_1): Counting attached table input_function_table_2 to database test_1
2026.05.06 08:48:35.841954 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.input_function_table_2' with status OK
2026.05.06 08:48:35.842002 [ 32339 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.842067 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.t' in BackgroundLoad
2026.05.06 08:48:35.842084 [ 32339 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.842118 [ 32397 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.t
2026.05.06 08:48:35.842138 [ 32339 ] {} <Test> DatabaseAtomic (test_15): Counting attached table t2 to database test_15
2026.05.06 08:48:35.842294 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.t2' with status OK
2026.05.06 08:48:35.842354 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.dist_01781' in BackgroundLoad
2026.05.06 08:48:35.842396 [ 32339 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.dist_01781
2026.05.06 08:48:35.842424 [ 32395 ] {} <Debug> DatabaseAtomic (01778_db): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.842475 [ 32395 ] {} <Debug> DatabaseAtomic (01778_db): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.842529 [ 32395 ] {} <Test> DatabaseAtomic (01778_db): Counting attached table hierarchy_source_table to database 01778_db
2026.05.06 08:48:35.842617 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table 01778_db.hierarchy_source_table' with status OK
2026.05.06 08:48:35.842719 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.set_index_not__fuzz_0' in BackgroundLoad
2026.05.06 08:48:35.842800 [ 32395 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.set_index_not__fuzz_0
2026.05.06 08:48:35.843116 [ 32438 ] {} <Test> test_13.collapsing_table (94c5b861-aaff-4d33-97f7-126faaa149e6): loadDataPart: inserting all_1_1_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.843178 [ 32438 ] {} <Trace> test_13.collapsing_table (94c5b861-aaff-4d33-97f7-126faaa149e6): Finished loading Active part all_1_1_1 on disk default
2026.05.06 08:48:35.843304 [ 32398 ] {} <Debug> test_13.collapsing_table (94c5b861-aaff-4d33-97f7-126faaa149e6): Loaded data parts (1 items)
2026.05.06 08:48:35.843683 [ 32432 ] {} <Test> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): loadDataPart: inserting 20190302_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.843769 [ 32432 ] {} <Trace> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): Finished loading Active part 20190302_2_2_0 on disk default
2026.05.06 08:48:35.843914 [ 32381 ] {} <Debug> test_13.mt (edc20eba-2b6e-45f2-9fec-ee660dc32083): Loaded data parts (3 items)
2026.05.06 08:48:35.844071 [ 32403 ] {} <Debug> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Loading data parts
2026.05.06 08:48:35.844373 [ 32390 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.844441 [ 32390 ] {} <Test> DatabaseAtomic (test_13): Counting attached table date_t to database test_13
2026.05.06 08:48:35.844519 [ 32402 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.844577 [ 32402 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.844594 [ 32390 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.date_t' with status OK
2026.05.06 08:48:35.845009 [ 32402 ] {} <Test> DatabaseAtomic (test_13): Counting attached table mv_source to database test_13
2026.05.06 08:48:35.845102 [ 32398 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.845300 [ 32398 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.845311 [ 32390 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.table_src' in BackgroundLoad
2026.05.06 08:48:35.845375 [ 32398 ] {} <Test> DatabaseAtomic (test_13): Counting attached table collapsing_table to database test_13
2026.05.06 08:48:35.845382 [ 32390 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.table_src
2026.05.06 08:48:35.845453 [ 32381 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.845478 [ 32402 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.mv_source' with status OK
2026.05.06 08:48:35.845521 [ 32381 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.845563 [ 32402 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.mt_00763_1' in BackgroundLoad
2026.05.06 08:48:35.845596 [ 32381 ] {} <Test> DatabaseAtomic (test_13): Counting attached table mt to database test_13
2026.05.06 08:48:35.845617 [ 32402 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.mt_00763_1
2026.05.06 08:48:35.845742 [ 32398 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.collapsing_table' with status OK
2026.05.06 08:48:35.847576 [ 32403 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.847653 [ 32403 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.848121 [ 32370 ] {} <Debug> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading data parts
2026.05.06 08:48:35.848156 [ 32398 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.tab1' in BackgroundLoad
2026.05.06 08:48:35.848221 [ 32398 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.tab1
2026.05.06 08:48:35.848262 [ 32435 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Loading Active part all_1_3_3 from disk default
2026.05.06 08:48:35.848350 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.mt' with status OK
2026.05.06 08:48:35.848430 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test.avro' in BackgroundLoad
2026.05.06 08:48:35.848475 [ 32381 ] {} <Trace> DatabaseAtomic (test): Loading table test.avro
2026.05.06 08:48:35.848558 [ 32371 ] {} <Debug> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Loading data parts
2026.05.06 08:48:35.848668 [ 32353 ] {} <Debug> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): Loading data parts
2026.05.06 08:48:35.848891 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk 03008_s3_plain_rewritable eligible for search: false (mode ANY)
2026.05.06 08:48:35.848913 [ 32370 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.848945 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk azure eligible for search: true (mode ANY)
2026.05.06 08:48:35.848972 [ 32370 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Will load 11 parts using up to 64 threads
2026.05.06 08:48:35.849010 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk backups eligible for search: true (mode ANY)
2026.05.06 08:48:35.849063 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk azure eligible for search: true (mode ANY)
2026.05.06 08:48:35.849094 [ 32442 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading Active part 9_0_0_0 from disk default
2026.05.06 08:48:35.849256 [ 32387 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading Active part 2_0_0_0 from disk default
2026.05.06 08:48:35.849281 [ 32424 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading Active part 7_0_0_0 from disk default
2026.05.06 08:48:35.850038 [ 32426 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading Active part 3_2_2_0 from disk default
2026.05.06 08:48:35.850093 [ 32418 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading Active part 5_2_2_0 from disk default
2026.05.06 08:48:35.850135 [ 32342 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading Active part 1_0_0_0 from disk default
2026.05.06 08:48:35.850170 [ 32351 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading Active part 0_0_0_0 from disk default
2026.05.06 08:48:35.850205 [ 32388 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading Active part 4_0_0_0 from disk default
2026.05.06 08:48:35.850221 [ 32337 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading Active part 3_3_3_0 from disk default
2026.05.06 08:48:35.850234 [ 32425 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading Active part 6_0_0_0 from disk default
2026.05.06 08:48:35.850187 [ 32376 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loading Active part 8_0_0_0 from disk default
2026.05.06 08:48:35.849105 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk default eligible for search: true (mode ANY)
2026.05.06 08:48:35.851422 [ 32395 ] {} <Debug> test_13.set_index_not__fuzz_0 (2b581513-aa4c-4105-9ef7-173d47f5a259): Loading data parts
2026.05.06 08:48:35.852144 [ 32371 ] {} <Test> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.852204 [ 32371 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Will load 2 parts using up to 64 threads
2026.05.06 08:48:35.852214 [ 32435 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): loadDataPart: inserting all_1_3_3 (state Active) into data_parts_indexes
2026.05.06 08:48:35.852270 [ 32435 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Finished loading Active part all_1_3_3 on disk default
2026.05.06 08:48:35.852345 [ 32434 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Loading Active part 20191002_0_0_0_4 from disk default
2026.05.06 08:48:35.852395 [ 32442 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataPart: inserting 9_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.852413 [ 32403 ] {} <Debug> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Found 5 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:35.852447 [ 32442 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Finished loading Active part 9_0_0_0 on disk default
2026.05.06 08:48:35.852493 [ 32403 ] {} <Debug> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Loaded data parts (1 items)
2026.05.06 08:48:35.852497 [ 32424 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataPart: inserting 7_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.852569 [ 32424 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Finished loading Active part 7_0_0_0 on disk default
2026.05.06 08:48:35.852647 [ 32387 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataPart: inserting 2_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.852711 [ 32387 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Finished loading Active part 2_0_0_0 on disk default
2026.05.06 08:48:35.852863 [ 32403 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.852905 [ 32403 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.852958 [ 32403 ] {} <Test> DatabaseAtomic (test_1): Counting attached table table_with_column_ttl to database test_1
2026.05.06 08:48:35.853081 [ 32403 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.table_with_column_ttl' with status OK
2026.05.06 08:48:35.853122 [ 32426 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataPart: inserting 3_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.853140 [ 32403 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.alter_default' in BackgroundLoad
2026.05.06 08:48:35.853167 [ 32426 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Finished loading Active part 3_2_2_0 on disk default
2026.05.06 08:48:35.853192 [ 32403 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.alter_default
2026.05.06 08:48:35.853419 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading data parts
2026.05.06 08:48:35.853496 [ 32418 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataPart: inserting 5_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.853557 [ 32418 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Finished loading Active part 5_2_2_0 on disk default
2026.05.06 08:48:35.853702 [ 32398 ] {} <Debug> test_1.tab1 (49273954-5371-402b-9d23-2b07f07e9b78): Loading data parts
2026.05.06 08:48:35.853913 [ 32390 ] {} <Debug> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Loading data parts
2026.05.06 08:48:35.854260 [ 32398 ] {} <Test> test_1.tab1 (49273954-5371-402b-9d23-2b07f07e9b78): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.854331 [ 32398 ] {} <Trace> test_1.tab1 (49273954-5371-402b-9d23-2b07f07e9b78): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.854376 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk disk1_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.854476 [ 32428 ] {} <Trace> test_1.tab1 (49273954-5371-402b-9d23-2b07f07e9b78): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.854515 [ 32390 ] {} <Test> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.854751 [ 32343 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Loading data parts
2026.05.06 08:48:35.854762 [ 32395 ] {} <Test> test_13.set_index_not__fuzz_0 (2b581513-aa4c-4105-9ef7-173d47f5a259): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.854801 [ 32395 ] {} <Trace> test_13.set_index_not__fuzz_0 (2b581513-aa4c-4105-9ef7-173d47f5a259): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.856275 [ 32434 ] {} <Test> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): loadDataPart: inserting 20191002_0_0_0_4 (state Active) into data_parts_indexes
2026.05.06 08:48:35.856344 [ 32434 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Finished loading Active part 20191002_0_0_0_4 on disk default
2026.05.06 08:48:35.856955 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk disk2_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.857026 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk disk3_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.857108 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk disk4_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.857158 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.857194 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.857228 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk local_disk_3 eligible for search: true (mode ANY)
2026.05.06 08:48:35.857275 [ 32434 ] {} <Trace> test_13.set_index_not__fuzz_0 (2b581513-aa4c-4105-9ef7-173d47f5a259): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.858675 [ 32425 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataPart: inserting 6_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.858733 [ 32425 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Finished loading Active part 6_0_0_0 on disk default
2026.05.06 08:48:35.858746 [ 32351 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataPart: inserting 0_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.858798 [ 32351 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Finished loading Active part 0_0_0_0 on disk default
2026.05.06 08:48:35.859020 [ 32440 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Loading Active part 20191002_1_1_0_4 from disk default
2026.05.06 08:48:35.859127 [ 32337 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataPart: inserting 3_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.859178 [ 32337 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Finished loading Active part 3_3_3_0 on disk default
2026.05.06 08:48:35.854573 [ 32390 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Will load 8 parts using up to 64 threads
2026.05.06 08:48:35.859413 [ 32343 ] {} <Test> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.859490 [ 32343 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.859758 [ 32431 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.859904 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.859968 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk local_disk_2 eligible for search: true (mode ANY)
2026.05.06 08:48:35.859998 [ 32430 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Loading Active part all_0_0_1_1 from disk default
2026.05.06 08:48:35.860031 [ 32388 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataPart: inserting 4_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.860101 [ 32388 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Finished loading Active part 4_0_0_0 on disk default
2026.05.06 08:48:35.860347 [ 32397 ] {} <Debug> test_13.t (46de70a1-5dc9-4a0b-943b-c1f58cc5512f): Loading data parts
2026.05.06 08:48:35.860514 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk local_disk_3 eligible for search: true (mode ANY)
2026.05.06 08:48:35.860568 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.860604 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.860920 [ 32397 ] {} <Test> test_13.t (46de70a1-5dc9-4a0b-943b-c1f58cc5512f): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.860983 [ 32397 ] {} <Trace> test_13.t (46de70a1-5dc9-4a0b-943b-c1f58cc5512f): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.861112 [ 32402 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.861134 [ 32428 ] {} <Test> test_1.tab1 (49273954-5371-402b-9d23-2b07f07e9b78): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.861160 [ 32402 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.861181 [ 32428 ] {} <Trace> test_1.tab1 (49273954-5371-402b-9d23-2b07f07e9b78): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.861283 [ 32431 ] {} <Test> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.861295 [ 32398 ] {} <Debug> test_1.tab1 (49273954-5371-402b-9d23-2b07f07e9b78): Loaded data parts (1 items)
2026.05.06 08:48:35.861289 [ 32448 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Active part all_1_1_0_65 from disk default
2026.05.06 08:48:35.861345 [ 32431 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.861345 [ 32446 ] {} <Debug> StorageDistributed (dist_01781): Removing /var/lib/clickhouse/store/2de/2de80d8f-3922-4b72-9b15-9f8529979324/shard1_replica1 (used for async INSERT into Distributed)
2026.05.06 08:48:35.861634 [ 32376 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataPart: inserting 8_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.861686 [ 32446 ] {} <Debug> StorageDistributed (dist_01781): Removing /var/lib/clickhouse/store/2de/2de80d8f-3922-4b72-9b15-9f8529979324/shard2_replica1 (used for async INSERT into Distributed)
2026.05.06 08:48:35.861687 [ 32376 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Finished loading Active part 8_0_0_0 on disk default
2026.05.06 08:48:35.861890 [ 32398 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.861951 [ 32398 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.861985 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.862022 [ 32398 ] {} <Test> DatabaseAtomic (test_1): Counting attached table tab1 to database test_1
2026.05.06 08:48:35.862028 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.862110 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.862196 [ 32398 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.tab1' with status OK
2026.05.06 08:48:35.862232 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.862270 [ 32434 ] {} <Test> test_13.set_index_not__fuzz_0 (2b581513-aa4c-4105-9ef7-173d47f5a259): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.862284 [ 32398 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.t1' in BackgroundLoad
2026.05.06 08:48:35.862294 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk s3_disk_02944 eligible for search: true (mode ANY)
2026.05.06 08:48:35.862275 [ 32339 ] {} <Debug> StorageDistributed (dist_01781): Auto-increment is 0
2026.05.06 08:48:35.862333 [ 32434 ] {} <Trace> test_13.set_index_not__fuzz_0 (2b581513-aa4c-4105-9ef7-173d47f5a259): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.862356 [ 32398 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.t1
2026.05.06 08:48:35.862367 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk s3_disk_02963 eligible for search: true (mode ANY)
2026.05.06 08:48:35.862644 [ 32395 ] {} <Debug> test_13.set_index_not__fuzz_0 (2b581513-aa4c-4105-9ef7-173d47f5a259): Loaded data parts (1 items)
2026.05.06 08:48:35.862704 [ 32340 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.862756 [ 32340 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Will load 2 parts using up to 64 threads
2026.05.06 08:48:35.862797 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk s3_no_cache eligible for search: true (mode ANY)
2026.05.06 08:48:35.862876 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): is disk s3_plain_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.862953 [ 32450 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Loading Active part all_0_0_0 from disk s3_disk
2026.05.06 08:48:35.862963 [ 32344 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Loading Active part all_1_1_0 from disk s3_disk
2026.05.06 08:48:35.862969 [ 32381 ] {} <Debug> DatabaseAtomic (test): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.863188 [ 32381 ] {} <Debug> DatabaseAtomic (test): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.863252 [ 32395 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.863258 [ 32381 ] {} <Test> DatabaseAtomic (test): Counting attached table avro to database test
2026.05.06 08:48:35.863324 [ 32342 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): loadDataPart: inserting 1_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.863312 [ 32395 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.863412 [ 32342 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Finished loading Active part 1_0_0_0 on disk default
2026.05.06 08:48:35.863421 [ 32395 ] {} <Test> DatabaseAtomic (test_13): Counting attached table set_index_not__fuzz_0 to database test_13
2026.05.06 08:48:35.863461 [ 32339 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.863573 [ 32440 ] {} <Test> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): loadDataPart: inserting 20191002_1_1_0_4 (state Active) into data_parts_indexes
2026.05.06 08:48:35.863645 [ 32440 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Finished loading Active part 20191002_1_1_0_4 on disk default
2026.05.06 08:48:35.864286 [ 32448 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_65 (state Active) into data_parts_indexes
2026.05.06 08:48:35.864332 [ 32448 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Active part all_1_1_0_65 on disk default
2026.05.06 08:48:35.864345 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test.avro' with status OK
2026.05.06 08:48:35.864578 [ 32339 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.864660 [ 32370 ] {} <Debug> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Loaded data parts (11 items)
2026.05.06 08:48:35.864693 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.view_4' in BackgroundLoad
2026.05.06 08:48:35.864717 [ 32339 ] {} <Test> DatabaseAtomic (test_1): Counting attached table dist_01781 to database test_1
2026.05.06 08:48:35.864728 [ 32401 ] {} <Debug> test_13.x (9c9f9c33-482c-41af-b377-a5df18c6d3ec): Loading data parts
2026.05.06 08:48:35.864767 [ 32370 ] {} <Information> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.864783 [ 32381 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.view_4
2026.05.06 08:48:35.864939 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.set_index_not__fuzz_0' with status OK
2026.05.06 08:48:35.865051 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_xfltqef0.mt' in BackgroundLoad
2026.05.06 08:48:35.865116 [ 32395 ] {} <Trace> DatabaseAtomic (test_xfltqef0): Loading table test_xfltqef0.mt
2026.05.06 08:48:35.865203 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.dist_01781' with status OK
2026.05.06 08:48:35.865256 [ 32370 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.865260 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.add_aggregate' in BackgroundLoad
2026.05.06 08:48:35.865314 [ 32370 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.865329 [ 32339 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.add_aggregate
2026.05.06 08:48:35.865371 [ 32370 ] {} <Test> DatabaseAtomic (test_13): Counting attached table table_for_freeze_replicated to database test_13
2026.05.06 08:48:35.865441 [ 31633 ] {} <Debug> ConfigProcessor: Processing configuration file '/etc/clickhouse-server/decimals_dictionary.xml'.
2026.05.06 08:48:35.865545 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.table_for_freeze_replicated' with status OK
2026.05.06 08:48:35.865690 [ 32339 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.865697 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.xp' in BackgroundLoad
2026.05.06 08:48:35.865743 [ 32339 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.865730 [ 32450 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0/columns.txt)
2026.05.06 08:48:35.865730 [ 32344 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0/columns.txt)
2026.05.06 08:48:35.865800 [ 32339 ] {} <Test> DatabaseAtomic (test_1): Counting attached table add_aggregate to database test_1
2026.05.06 08:48:35.865805 [ 32370 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.xp
2026.05.06 08:48:35.865879 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.add_aggregate' with status OK
2026.05.06 08:48:35.865933 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table 01945.db.test_dictionary_values' in BackgroundLoad
2026.05.06 08:48:35.865975 [ 32339 ] {} <Trace> DatabaseAtomic (01945.db): Loading table 01945.db.test_dictionary_values
2026.05.06 08:48:35.865991 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:35.865990 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:35.866231 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:35.866232 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:35.866505 [ 32339 ] {} <Debug> DatabaseAtomic (01945.db): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.866532 [ 32403 ] {} <Debug> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Loading data parts
2026.05.06 08:48:35.866546 [ 32339 ] {} <Debug> DatabaseAtomic (01945.db): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.866530 [ 32430 ] {} <Test> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): loadDataPart: inserting all_0_0_1_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.866593 [ 32339 ] {} <Test> DatabaseAtomic (01945.db): Counting attached table test_dictionary_values to database 01945.db
2026.05.06 08:48:35.866580 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.866609 [ 32430 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Finished loading Active part all_0_0_1_1 on disk default
2026.05.06 08:48:35.866580 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.866685 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table 01945.db.test_dictionary_values' with status OK
2026.05.06 08:48:35.866718 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -67, current is 2771
2026.05.06 08:48:35.866748 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.mv_target' in BackgroundLoad
2026.05.06 08:48:35.866790 [ 32339 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.mv_target
2026.05.06 08:48:35.866838 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: 4f35d3dc379904cf70c62240c71a85ac, offset: 0, size: 67
2026.05.06 08:48:35.866898 [ 32403 ] {} <Test> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.866864 [ 32343 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Found 2 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:35.866960 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 67, current is 1
2026.05.06 08:48:35.866939 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.866931 [ 32371 ] {} <Debug> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Loaded data parts (2 items)
2026.05.06 08:48:35.866944 [ 32403 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.866932 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Found 18 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:35.867087 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: 4f35d3dc379904cf70c62240c71a85ac, offset: 0, size: 67
2026.05.06 08:48:35.867148 [ 32371 ] {} <Information> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.867203 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loaded data parts (1 items)
2026.05.06 08:48:35.867255 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.867320 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:35.867353 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:35.867520 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:35.867521 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:35.867587 [ 32371 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.867648 [ 32371 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.867712 [ 32371 ] {} <Test> DatabaseAtomic (test_1): Counting attached table replicated_mutation_table to database test_1
2026.05.06 08:48:35.867845 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:35.867869 [ 32429 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.867871 [ 32433 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Loading Active part all_6_6_0 from disk default
2026.05.06 08:48:35.867846 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:35.868312 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_57.txt entry, commands size: 1
2026.05.06 08:48:35.868646 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_50.txt entry, commands size: 1
2026.05.06 08:48:35.868950 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_65.txt entry, commands size: 1
2026.05.06 08:48:35.868980 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.869044 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.869071 [ 32374 ] {} <Test> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.869135 [ 32381 ] {} <Test> DatabaseAtomic (test_1): Counting attached table view_4 to database test_1
2026.05.06 08:48:35.869150 [ 32374 ] {} <Debug> test_1.test_s3_mt_dst (de4884f8-fc4a-4d02-a4fc-ab91a41c4d2b): There are no data parts
2026.05.06 08:48:35.869233 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_5.txt entry, commands size: 1
2026.05.06 08:48:35.869234 [ 32439 ] {} <Trace> test_13.t (46de70a1-5dc9-4a0b-943b-c1f58cc5512f): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.869221 [ 32344 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0/serialization.json)
2026.05.06 08:48:35.869235 [ 32450 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0/serialization.json)
2026.05.06 08:48:35.869528 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:35.869530 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:35.869560 [ 32433 ] {} <Test> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): loadDataPart: inserting all_6_6_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.869671 [ 32445 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Loading Active part all_3_3_0 from disk default
2026.05.06 08:48:35.869689 [ 32433 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Finished loading Active part all_6_6_0 on disk default
2026.05.06 08:48:35.869725 [ 32343 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Loaded data parts (1 items)
2026.05.06 08:48:35.869744 [ 32438 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Loading Active part all_5_5_0 from disk default
2026.05.06 08:48:35.869799 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:35.869798 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:35.869595 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_12.txt entry, commands size: 1
2026.05.06 08:48:35.869835 [ 32343 ] {} <Information> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.870182 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.870186 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.870201 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_15.txt entry, commands size: 1
2026.05.06 08:48:35.870323 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -205, current is 2704
2026.05.06 08:48:35.870432 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: 0f8745be7660054901be780588901782, offset: 0, size: 205
2026.05.06 08:48:35.870455 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.870530 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_49.txt entry, commands size: 1
2026.05.06 08:48:35.870545 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 205, current is 68
2026.05.06 08:48:35.870579 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.870618 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.870643 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: 0f8745be7660054901be780588901782, offset: 0, size: 205
2026.05.06 08:48:35.870661 [ 32343 ] {} <Test> DatabaseAtomic (test_1): Counting attached table t to database test_1
2026.05.06 08:48:35.870645 [ 32339 ] {} <Debug> test_13.mv_target (b8f18603-ee93-40fa-8db0-bef4b9018f92): Loading data parts
2026.05.06 08:48:35.870713 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:35.870774 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.870855 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_47.txt entry, commands size: 1
2026.05.06 08:48:35.870855 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:35.870894 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:35.870928 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:35.871140 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_55.txt entry, commands size: 1
2026.05.06 08:48:35.871132 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:35.871132 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:35.871438 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_62.txt entry, commands size: 1
2026.05.06 08:48:35.871494 [ 32438 ] {} <Test> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): loadDataPart: inserting all_5_5_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.871541 [ 32439 ] {} <Test> test_13.t (46de70a1-5dc9-4a0b-943b-c1f58cc5512f): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.871564 [ 32438 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Finished loading Active part all_5_5_0 on disk default
2026.05.06 08:48:35.871576 [ 32445 ] {} <Test> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): loadDataPart: inserting all_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.871583 [ 32439 ] {} <Trace> test_13.t (46de70a1-5dc9-4a0b-943b-c1f58cc5512f): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.871681 [ 32445 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Finished loading Active part all_3_3_0 on disk default
2026.05.06 08:48:35.871649 [ 32353 ] {} <Test> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.871714 [ 32436 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Loading Active part all_0_0_0_3 from disk default
2026.05.06 08:48:35.871770 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_14.txt entry, commands size: 1
2026.05.06 08:48:35.871782 [ 32353 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.871793 [ 32397 ] {} <Debug> test_13.t (46de70a1-5dc9-4a0b-943b-c1f58cc5512f): Loaded data parts (1 items)
2026.05.06 08:48:35.871897 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.replicated_mutation_table' with status OK
2026.05.06 08:48:35.871975 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.select_final' in BackgroundLoad
2026.05.06 08:48:35.872035 [ 32371 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.select_final
2026.05.06 08:48:35.872058 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_21.txt entry, commands size: 1
2026.05.06 08:48:35.872277 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.872280 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.view_4' with status OK
2026.05.06 08:48:35.872326 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.872351 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_10.txt entry, commands size: 1
2026.05.06 08:48:35.872475 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_jbjeaal4_1.t' in BackgroundLoad
2026.05.06 08:48:35.872479 [ 32397 ] {} <Test> DatabaseAtomic (test_13): Counting attached table t to database test_13
2026.05.06 08:48:35.872466 [ 32423 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): Loading Active part 0_1_1_1 from disk 03008_s3_plain_rewritable
2026.05.06 08:48:35.872543 [ 32381 ] {} <Trace> DatabaseAtomic (test_jbjeaal4_1): Loading table test_jbjeaal4_1.t
2026.05.06 08:48:35.872567 [ 32395 ] {} <Debug> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): Loading data parts
2026.05.06 08:48:35.872621 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.t' with status OK
2026.05.06 08:48:35.872646 [ 32450 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0/metadata_version.txt)
2026.05.06 08:48:35.872646 [ 32344 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0/metadata_version.txt)
2026.05.06 08:48:35.872676 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.insert_dedup_token2' in BackgroundLoad
2026.05.06 08:48:35.872724 [ 32374 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:35.872764 [ 32397 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.insert_dedup_token2
2026.05.06 08:48:35.872757 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_52.txt entry, commands size: 1
2026.05.06 08:48:35.872885 [ 32381 ] {} <Debug> DatabaseAtomic (test_jbjeaal4_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.872906 [ 32395 ] {} <Test> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.872921 [ 32381 ] {} <Debug> DatabaseAtomic (test_jbjeaal4_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.872921 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:35.872921 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:35.872970 [ 32395 ] {} <Trace> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): Will load 3 parts using up to 64 threads
2026.05.06 08:48:35.872965 [ 32381 ] {} <Test> DatabaseAtomic (test_jbjeaal4_1): Counting attached table t to database test_jbjeaal4_1
2026.05.06 08:48:35.873033 [ 32343 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.t' with status OK
2026.05.06 08:48:35.873097 [ 32343 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.insert_dedup_token1' in BackgroundLoad
2026.05.06 08:48:35.873126 [ 32442 ] {} <Trace> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): Loading Active part 20190302_2_2_0 from disk default
2026.05.06 08:48:35.873142 [ 32343 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.insert_dedup_token1
2026.05.06 08:48:35.873149 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:35.873155 [ 32398 ] {} <Debug> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): Loading data parts
2026.05.06 08:48:35.873193 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_jbjeaal4_1.t' with status OK
2026.05.06 08:48:35.873197 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_44.txt entry, commands size: 1
2026.05.06 08:48:35.873149 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:35.873351 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.02952_disjunction_optimization' in BackgroundLoad
2026.05.06 08:48:35.873392 [ 32381 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.02952_disjunction_optimization
2026.05.06 08:48:35.873444 [ 32429 ] {} <Test> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.873470 [ 32374 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: ::1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:35 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE728ED67EE14
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:35.873512 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.873513 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.873510 [ 32429 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.873646 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -1, current is 2499
2026.05.06 08:48:35.873667 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_16.txt entry, commands size: 1
2026.05.06 08:48:35.873672 [ 32429 ] {} <Trace> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): Loading Active part 20190301_1_1_0 from disk default
2026.05.06 08:48:35.873677 [ 32381 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.873750 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: c398d2bbee132ab3d078997de3f82f98, offset: 0, size: 1
2026.05.06 08:48:35.873763 [ 32381 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.873809 [ 32381 ] {} <Test> DatabaseAtomic (test_13): Counting attached table 02952_disjunction_optimization to database test_13
2026.05.06 08:48:35.873859 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 1, current is 273
2026.05.06 08:48:35.873860 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.873890 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.02952_disjunction_optimization' with status OK
2026.05.06 08:48:35.873945 [ 32436 ] {} <Test> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): loadDataPart: inserting all_0_0_0_3 (state Active) into data_parts_indexes
2026.05.06 08:48:35.873962 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: c398d2bbee132ab3d078997de3f82f98, offset: 0, size: 1
2026.05.06 08:48:35.873989 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_30.txt entry, commands size: 1
2026.05.06 08:48:35.874002 [ 32436 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Finished loading Active part all_0_0_0_3 on disk default
2026.05.06 08:48:35.874000 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.input_function_table_6' in BackgroundLoad
2026.05.06 08:48:35.874055 [ 32381 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.input_function_table_6
2026.05.06 08:48:35.874063 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.874114 [ 32403 ] {} <Debug> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Loaded data parts (1 items)
2026.05.06 08:48:35.874179 [ 32403 ] {} <Information> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.874230 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:35.874231 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:35.874280 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_42.txt entry, commands size: 1
2026.05.06 08:48:35.874347 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.874384 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.874410 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:35.874417 [ 32374 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:35.874410 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:35.874467 [ 32381 ] {} <Test> DatabaseAtomic (test_1): Counting attached table input_function_table_6 to database test_1
2026.05.06 08:48:35.874545 [ 32403 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.874577 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_60.txt entry, commands size: 1
2026.05.06 08:48:35.874595 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.input_function_table_6' with status OK
2026.05.06 08:48:35.874647 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.test_table' in BackgroundLoad
2026.05.06 08:48:35.874681 [ 32381 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.test_table
2026.05.06 08:48:35.874681 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:35.874681 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:35.874861 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_29.txt entry, commands size: 1
2026.05.06 08:48:35.874988 [ 32370 ] {} <Debug> test_13.xp (6327e17b-0bcc-4830-9041-31a1aa4ea77e): Loading data parts
2026.05.06 08:48:35.875144 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_35.txt entry, commands size: 1
2026.05.06 08:48:35.874582 [ 32403 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.875559 [ 32403 ] {} <Test> DatabaseAtomic (test_13): Counting attached table alter_default to database test_13
2026.05.06 08:48:35.875744 [ 32403 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.alter_default' with status OK
2026.05.06 08:48:35.875881 [ 32403 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.bug_13492' in BackgroundLoad
2026.05.06 08:48:35.875926 [ 32403 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.bug_13492
2026.05.06 08:48:35.876088 [ 32432 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Loading Active part all_4_4_0 from disk default
2026.05.06 08:48:35.876101 [ 32444 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Loading Active part all_7_7_0 from disk default
2026.05.06 08:48:35.876153 [ 32450 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vtq/qvasrnsxfrgvretnklcxwjzfgekce (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0/checksums.txt)
2026.05.06 08:48:35.876154 [ 32344 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kuy/tqhznrtdkduuwzhapdvrhykjitatq (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0/checksums.txt)
2026.05.06 08:48:35.876286 [ 32442 ] {} <Test> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): loadDataPart: inserting 20190302_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.876330 [ 32442 ] {} <Trace> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): Finished loading Active part 20190302_2_2_0 on disk default
2026.05.06 08:48:35.876381 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(3606a251266710571c29f7d537f11077): Reset state for seek to position 0
2026.05.06 08:48:35.876381 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(349af8778c2543613ccc71c8f0f56bd8): Reset state for seek to position 0
2026.05.06 08:48:35.876425 [ 32442 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Loading Active part all_8_8_0 from disk default
2026.05.06 08:48:35.876579 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(349af8778c2543613ccc71c8f0f56bd8): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:35.876580 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(3606a251266710571c29f7d537f11077): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:35.872774 [ 32401 ] {} <Test> test_13.x (9c9f9c33-482c-41af-b377-a5df18c6d3ec): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.876912 [ 32429 ] {} <Test> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): loadDataPart: inserting 20190301_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.876934 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(349af8778c2543613ccc71c8f0f56bd8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 349af8778c2543613ccc71c8f0f56bd8, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.876942 [ 32401 ] {} <Trace> test_13.x (9c9f9c33-482c-41af-b377-a5df18c6d3ec): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.876934 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(3606a251266710571c29f7d537f11077): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 3606a251266710571c29f7d537f11077, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.877037 [ 32398 ] {} <Test> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.877092 [ 32377 ] {} <Trace> test_13.x (9c9f9c33-482c-41af-b377-a5df18c6d3ec): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.876963 [ 32429 ] {} <Trace> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): Finished loading Active part 20190301_1_1_0 on disk default
2026.05.06 08:48:35.877098 [ 32398 ] {} <Trace> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): Will load 2 parts using up to 64 threads
2026.05.06 08:48:35.877593 [ 32432 ] {} <Test> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): loadDataPart: inserting all_4_4_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.877642 [ 32432 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Finished loading Active part all_4_4_0 on disk default
2026.05.06 08:48:35.877716 [ 32442 ] {} <Test> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): loadDataPart: inserting all_8_8_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.877760 [ 32442 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Finished loading Active part all_8_8_0 on disk default
2026.05.06 08:48:35.877783 [ 32444 ] {} <Test> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): loadDataPart: inserting all_7_7_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.877836 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_38.txt entry, commands size: 1
2026.05.06 08:48:35.877862 [ 32432 ] {} <Trace> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): Loading Active part 2_5_5_0 from disk default
2026.05.06 08:48:35.877861 [ 32444 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Finished loading Active part all_7_7_0 on disk default
2026.05.06 08:48:35.878078 [ 32390 ] {} <Debug> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Loaded data parts (8 items)
2026.05.06 08:48:35.878186 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_36.txt entry, commands size: 1
2026.05.06 08:48:35.878514 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_40.txt entry, commands size: 1
2026.05.06 08:48:35.878735 [ 31633 ] {} <Debug> ConfigProcessor: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/decimals_dictionary.xml'.
2026.05.06 08:48:35.878812 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_48.txt entry, commands size: 1
2026.05.06 08:48:35.878873 [ 32390 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.878971 [ 32339 ] {} <Test> test_13.mv_target (b8f18603-ee93-40fa-8db0-bef4b9018f92): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.879001 [ 32390 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.879020 [ 32339 ] {} <Trace> test_13.mv_target (b8f18603-ee93-40fa-8db0-bef4b9018f92): Will load 2 parts using up to 64 threads
2026.05.06 08:48:35.879022 [ 32371 ] {} <Debug> test_1.select_final (fa9994ba-3ee9-423e-8437-49e256c1948e): Loading data parts
2026.05.06 08:48:35.879083 [ 32390 ] {} <Test> DatabaseAtomic (test_1): Counting attached table table_src to database test_1
2026.05.06 08:48:35.879136 [ 32425 ] {} <Trace> test_13.mv_target (b8f18603-ee93-40fa-8db0-bef4b9018f92): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.879252 [ 32390 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.table_src' with status OK
2026.05.06 08:48:35.879251 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -276, current is 2498
2026.05.06 08:48:35.879280 [ 32371 ] {} <Test> test_1.select_final (fa9994ba-3ee9-423e-8437-49e256c1948e): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.879344 [ 32390 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.dynamic_test_2' in BackgroundLoad
2026.05.06 08:48:35.879221 [ 32453 ] {} <Trace> test_13.mv_target (b8f18603-ee93-40fa-8db0-bef4b9018f92): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.879363 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: 3606a251266710571c29f7d537f11077, offset: 0, size: 276
2026.05.06 08:48:35.879401 [ 32390 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.dynamic_test_2
2026.05.06 08:48:35.879478 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 276, current is 274
2026.05.06 08:48:35.879572 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: 3606a251266710571c29f7d537f11077, offset: 0, size: 276
2026.05.06 08:48:35.879359 [ 32371 ] {} <Debug> test_1.select_final (fa9994ba-3ee9-423e-8437-49e256c1948e): There are no data parts
2026.05.06 08:48:35.879653 [ 32432 ] {} <Test> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): loadDataPart: inserting 2_5_5_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.879763 [ 32432 ] {} <Trace> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): Finished loading Active part 2_5_5_0 on disk default
2026.05.06 08:48:35.879794 [ 31633 ] {} <Trace> ExternalDictionariesLoader: Loading config file '/etc/clickhouse-server/executable_dictionary.xml'.
2026.05.06 08:48:35.879833 [ 32390 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.879873 [ 32390 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.879916 [ 32390 ] {} <Test> DatabaseAtomic (test_1): Counting attached table dynamic_test_2 to database test_1
2026.05.06 08:48:35.880004 [ 32390 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.dynamic_test_2' with status OK
2026.05.06 08:48:35.880055 [ 32390 ] {} <Debug> AsyncLoader: Execute load job 'load table shard_1.from_1' in BackgroundLoad
2026.05.06 08:48:35.880090 [ 32390 ] {} <Trace> DatabaseAtomic (shard_1): Loading table shard_1.from_1
2026.05.06 08:48:35.880212 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(3606a251266710571c29f7d537f11077): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 3606a251266710571c29f7d537f11077, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.880286 [ 32371 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.880338 [ 32371 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.880388 [ 32371 ] {} <Test> DatabaseAtomic (test_1): Counting attached table select_final to database test_1
2026.05.06 08:48:35.880412 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(3606a251266710571c29f7d537f11077): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:35.880467 [ 32381 ] {} <Debug> test_1.test_table (6eab3780-9bf8-422e-92c1-cfb5f73bab04): Loading data parts
2026.05.06 08:48:35.880515 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(3606a251266710571c29f7d537f11077): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:35.880528 [ 32370 ] {} <Test> test_13.xp (6327e17b-0bcc-4830-9041-31a1aa4ea77e): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.880523 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.select_final' with status OK
2026.05.06 08:48:35.880591 [ 32370 ] {} <Trace> test_13.xp (6327e17b-0bcc-4830-9041-31a1aa4ea77e): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.880645 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.t_modify_to_nullable' in BackgroundLoad
2026.05.06 08:48:35.880690 [ 32371 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.t_modify_to_nullable
2026.05.06 08:48:35.880724 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(3606a251266710571c29f7d537f11077): Key: 3606a251266710571c29f7d537f11077. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:35.880765 [ 32374 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:35.880766 [ 32351 ] {} <Trace> test_13.xp (6327e17b-0bcc-4830-9041-31a1aa4ea77e): Loading Active part 202001_1_1_0 from disk default
2026.05.06 08:48:35.880809 [ 32425 ] {} <Test> test_13.mv_target (b8f18603-ee93-40fa-8db0-bef4b9018f92): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.880785 [ 32419 ] {} <Trace> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): Loading Active part 1_1_1_0 from disk default
2026.05.06 08:48:35.880860 [ 32425 ] {} <Trace> test_13.mv_target (b8f18603-ee93-40fa-8db0-bef4b9018f92): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.881066 [ 32381 ] {} <Test> test_1.test_table (6eab3780-9bf8-422e-92c1-cfb5f73bab04): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.881098 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(349af8778c2543613ccc71c8f0f56bd8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 349af8778c2543613ccc71c8f0f56bd8, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.881127 [ 32381 ] {} <Trace> test_1.test_table (6eab3780-9bf8-422e-92c1-cfb5f73bab04): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.881198 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(349af8778c2543613ccc71c8f0f56bd8): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:35.881219 [ 32374 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:35 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE728EDD7BC6E
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:35.881323 [ 32443 ] {} <Trace> test_1.test_table (6eab3780-9bf8-422e-92c1-cfb5f73bab04): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.881691 [ 32374 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:35.881879 [ 32453 ] {} <Test> test_13.mv_target (b8f18603-ee93-40fa-8db0-bef4b9018f92): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.881940 [ 32453 ] {} <Trace> test_13.mv_target (b8f18603-ee93-40fa-8db0-bef4b9018f92): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.881938 [ 32344 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0/n.cmrk2)
2026.05.06 08:48:35.882087 [ 32339 ] {} <Debug> test_13.mv_target (b8f18603-ee93-40fa-8db0-bef4b9018f92): Loaded data parts (2 items)
2026.05.06 08:48:35.882105 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:35.882301 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:35.882531 [ 32339 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.882533 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.882582 [ 32339 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.882632 [ 32339 ] {} <Test> DatabaseAtomic (test_13): Counting attached table mv_target to database test_13
2026.05.06 08:48:35.882640 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -47, current is 2222
2026.05.06 08:48:35.882680 [ 32351 ] {} <Test> test_13.xp (6327e17b-0bcc-4830-9041-31a1aa4ea77e): loadDataPart: inserting 202001_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.882702 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: 33385792238d709413a230095c6af1d8, offset: 0, size: 47
2026.05.06 08:48:35.882725 [ 32351 ] {} <Trace> test_13.xp (6327e17b-0bcc-4830-9041-31a1aa4ea77e): Finished loading Active part 202001_1_1_0 on disk default
2026.05.06 08:48:35.882738 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.mv_target' with status OK
2026.05.06 08:48:35.882765 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 47, current is 550
2026.05.06 08:48:35.882795 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.json_columns' in BackgroundLoad
2026.05.06 08:48:35.882829 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: 33385792238d709413a230095c6af1d8, offset: 0, size: 47
2026.05.06 08:48:35.882879 [ 32339 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.json_columns
2026.05.06 08:48:35.882924 [ 32370 ] {} <Debug> test_13.xp (6327e17b-0bcc-4830-9041-31a1aa4ea77e): Loaded data parts (1 items)
2026.05.06 08:48:35.882978 [ 32374 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.883019 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.883035 [ 32374 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.883058 [ 32419 ] {} <Test> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): loadDataPart: inserting 1_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.883070 [ 32397 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Loading data parts
2026.05.06 08:48:35.883100 [ 32374 ] {} <Test> DatabaseAtomic (test_1): Counting attached table test_s3_mt_dst to database test_1
2026.05.06 08:48:35.883109 [ 32419 ] {} <Trace> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): Finished loading Active part 1_1_1_0 on disk default
2026.05.06 08:48:35.883155 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:35.883229 [ 32398 ] {} <Debug> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): Loaded data parts (2 items)
2026.05.06 08:48:35.883234 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:35.883267 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.test_s3_mt_dst' with status OK
2026.05.06 08:48:35.883277 [ 32403 ] {} <Debug> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading data parts
2026.05.06 08:48:35.883347 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.set' in BackgroundLoad
2026.05.06 08:48:35.883401 [ 32443 ] {} <Test> test_1.test_table (6eab3780-9bf8-422e-92c1-cfb5f73bab04): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.883389 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:35.883448 [ 32374 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.set
2026.05.06 08:48:35.883466 [ 32443 ] {} <Trace> test_1.test_table (6eab3780-9bf8-422e-92c1-cfb5f73bab04): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.883532 [ 32370 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.883448 [ 32397 ] {} <Test> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.883528 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(349af8778c2543613ccc71c8f0f56bd8): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:35.883637 [ 32397 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Will load 3 parts using up to 64 threads
2026.05.06 08:48:35.883671 [ 32381 ] {} <Debug> test_1.test_table (6eab3780-9bf8-422e-92c1-cfb5f73bab04): Loaded data parts (1 items)
2026.05.06 08:48:35.883707 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(349af8778c2543613ccc71c8f0f56bd8): Key: 349af8778c2543613ccc71c8f0f56bd8. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:35.883786 [ 32370 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.883836 [ 32370 ] {} <Test> DatabaseAtomic (test_13): Counting attached table xp to database test_13
2026.05.06 08:48:35.883982 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.xp' with status OK
2026.05.06 08:48:35.884044 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table rdb_test_13.t3' in BackgroundLoad
2026.05.06 08:48:35.884087 [ 32370 ] {} <Trace> DatabaseReplicated (rdb_test_13): Loading table rdb_test_13.t3
2026.05.06 08:48:35.884202 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.884332 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.884388 [ 32450 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0/n.cmrk2)
2026.05.06 08:48:35.884392 [ 32381 ] {} <Test> DatabaseAtomic (test_1): Counting attached table test_table to database test_1
2026.05.06 08:48:35.884429 [ 32344 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0/count.txt)
2026.05.06 08:48:35.884462 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:35.884543 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:35.884559 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.test_table' with status OK
2026.05.06 08:48:35.884576 [ 32370 ] {} <Debug> DatabaseReplicated (rdb_test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.884616 [ 32370 ] {} <Debug> DatabaseReplicated (rdb_test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.884611 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:35.884633 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.parallel_replicas_plain' in BackgroundLoad
2026.05.06 08:48:35.884656 [ 32370 ] {} <Test> DatabaseReplicated (rdb_test_13): Counting attached table t3 to database rdb_test_13
2026.05.06 08:48:35.884661 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.884686 [ 32381 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.parallel_replicas_plain
2026.05.06 08:48:35.884716 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -47, current is 2175
2026.05.06 08:48:35.884731 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table rdb_test_13.t3' with status OK
2026.05.06 08:48:35.884758 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: 3244d0979a74840152a85a31e615952a, offset: 0, size: 47
2026.05.06 08:48:35.884783 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.dst' in BackgroundLoad
2026.05.06 08:48:35.884801 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 47, current is 597
2026.05.06 08:48:35.884819 [ 32370 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.dst
2026.05.06 08:48:35.884821 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:35.884850 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: 3244d0979a74840152a85a31e615952a, offset: 0, size: 47
2026.05.06 08:48:35.884931 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.885018 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:35.885068 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:35.885107 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.885161 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:35.885168 [ 32378 ] {} <Trace> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): Loading Active part 20190303_3_3_0 from disk default
2026.05.06 08:48:35.885212 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -1, current is 2128
2026.05.06 08:48:35.885220 [ 32374 ] {} <Information> StorageSetOrJoinBase: Loaded from backup file store/534/5349a992-e61b-4e7e-9e3c-6fb6833c95d4/1.bin. 2 rows, 28.00 B. State has 2 unique rows.
2026.05.06 08:48:35.885259 [ 32398 ] {} <Debug> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): Loading mutation: mutation_3.txt entry, commands size: 1
2026.05.06 08:48:35.885298 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: d96b10f818af9fc56b70c3886e85f9c3, offset: 0, size: 1
2026.05.06 08:48:35.885369 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 1, current is 644
2026.05.06 08:48:35.885437 [ 32344 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: d96b10f818af9fc56b70c3886e85f9c3, offset: 0, size: 1
2026.05.06 08:48:35.885571 [ 32374 ] {} <Information> StorageSetOrJoinBase: Loaded from backup file store/534/5349a992-e61b-4e7e-9e3c-6fb6833c95d4/2.bin. 2 rows, 28.00 B. State has 2 unique rows.
2026.05.06 08:48:35.885606 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.885635 [ 32398 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.885665 [ 32450 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0/count.txt)
2026.05.06 08:48:35.885679 [ 32398 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.885722 [ 32398 ] {} <Test> DatabaseAtomic (test_1): Counting attached table t1 to database test_1
2026.05.06 08:48:35.885727 [ 32374 ] {} <Information> StorageSetOrJoinBase: Loaded from backup file store/534/5349a992-e61b-4e7e-9e3c-6fb6833c95d4/3.bin. 2 rows, 26.00 B. State has 3 unique rows.
2026.05.06 08:48:35.885735 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:35.885726 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:35.885775 [ 32371 ] {} <Debug> test_13.t_modify_to_nullable (cf6d4d1c-040f-4e41-9e38-934ae9036995): Loading data parts
2026.05.06 08:48:35.885829 [ 32398 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.t1' with status OK
2026.05.06 08:48:35.885825 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:35.885880 [ 32398 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.tab_00625' in BackgroundLoad
2026.05.06 08:48:35.885893 [ 32374 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.885915 [ 32398 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.tab_00625
2026.05.06 08:48:35.885987 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:35.886005 [ 32374 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.886064 [ 32374 ] {} <Test> DatabaseAtomic (test_1): Counting attached table set to database test_1
2026.05.06 08:48:35.886145 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.set' with status OK
2026.05.06 08:48:35.886197 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.input_function_table_5' in BackgroundLoad
2026.05.06 08:48:35.886233 [ 32374 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.input_function_table_5
2026.05.06 08:48:35.886233 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:35.886364 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.886412 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -1, current is 2127
2026.05.06 08:48:35.886449 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: eb35a528d8f8ea53396c56179f8107bc, offset: 0, size: 1
2026.05.06 08:48:35.886496 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 1, current is 645
2026.05.06 08:48:35.886532 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: eb35a528d8f8ea53396c56179f8107bc, offset: 0, size: 1
2026.05.06 08:48:35.886612 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.886619 [ 32374 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.886657 [ 32374 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.886683 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:35.886692 [ 32376 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.886699 [ 32374 ] {} <Test> DatabaseAtomic (test_13): Counting attached table input_function_table_5 to database test_13
2026.05.06 08:48:35.886682 [ 32421 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Loading Active part all_0_0_0 from disk default
2026.05.06 08:48:35.886744 [ 32446 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.886826 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.input_function_table_5' with status OK
2026.05.06 08:48:35.886840 [ 32377 ] {} <Test> test_13.x (9c9f9c33-482c-41af-b377-a5df18c6d3ec): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.886873 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.input_function_table_4' in BackgroundLoad
2026.05.06 08:48:35.886890 [ 32377 ] {} <Trace> test_13.x (9c9f9c33-482c-41af-b377-a5df18c6d3ec): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.886906 [ 32374 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.input_function_table_4
2026.05.06 08:48:35.887018 [ 32401 ] {} <Debug> test_13.x (9c9f9c33-482c-41af-b377-a5df18c6d3ec): Loaded data parts (1 items)
2026.05.06 08:48:35.887200 [ 32390 ] {} <Debug> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Loading data parts
2026.05.06 08:48:35.887227 [ 32374 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.887268 [ 32374 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.887317 [ 32374 ] {} <Test> DatabaseAtomic (test_1): Counting attached table input_function_table_4 to database test_1
2026.05.06 08:48:35.887399 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.input_function_table_4' with status OK
2026.05.06 08:48:35.887426 [ 32339 ] {} <Debug> test_13.json_columns (050ab0f2-ba47-4659-8212-6898acc8cf75): Loading data parts
2026.05.06 08:48:35.887469 [ 32390 ] {} <Test> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.887508 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table test_3.ordinary_distinct' in BackgroundLoad
2026.05.06 08:48:35.887521 [ 32390 ] {} <Debug> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): There are no data parts
2026.05.06 08:48:35.887535 [ 32401 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.887550 [ 32374 ] {} <Trace> DatabaseAtomic (test_3): Loading table test_3.ordinary_distinct
2026.05.06 08:48:35.887566 [ 32390 ] {} <Information> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.887588 [ 32401 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.887652 [ 32401 ] {} <Test> DatabaseAtomic (test_13): Counting attached table x to database test_13
2026.05.06 08:48:35.887816 [ 32401 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.x' with status OK
2026.05.06 08:48:35.887866 [ 32401 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.agg_func_col' in BackgroundLoad
2026.05.06 08:48:35.887901 [ 32401 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.agg_func_col
2026.05.06 08:48:35.887906 [ 32390 ] {} <Debug> DatabaseAtomic (shard_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.888008 [ 32390 ] {} <Debug> DatabaseAtomic (shard_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.888056 [ 32390 ] {} <Test> DatabaseAtomic (shard_1): Counting attached table from_1 to database shard_1
2026.05.06 08:48:35.888178 [ 32390 ] {} <Debug> AsyncLoader: Finish load job 'load table shard_1.from_1' with status OK
2026.05.06 08:48:35.888244 [ 32390 ] {} <Debug> AsyncLoader: Execute load job 'load table shard_1.data_02346' in BackgroundLoad
2026.05.06 08:48:35.888256 [ 32376 ] {} <Test> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.888285 [ 32390 ] {} <Trace> DatabaseAtomic (shard_1): Loading table shard_1.data_02346
2026.05.06 08:48:35.888319 [ 32376 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.888377 [ 32421 ] {} <Test> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): loadDataPart: inserting all_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.888419 [ 32378 ] {} <Test> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): loadDataPart: inserting 20190303_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.888445 [ 32421 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Finished loading Active part all_0_0_0 on disk default
2026.05.06 08:48:35.888465 [ 32378 ] {} <Trace> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): Finished loading Active part 20190303_3_3_0 on disk default
2026.05.06 08:48:35.888565 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:35.888591 [ 32423 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:35.888599 [ 32395 ] {} <Debug> test_xfltqef0.mt (5a56e9cb-c97d-43d6-8223-ba7d1d68f395): Loaded data parts (3 items)
2026.05.06 08:48:35.888662 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:35.888665 [ 32446 ] {} <Test> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.888736 [ 32446 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.888741 [ 32343 ] {} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Loading data parts
2026.05.06 08:48:35.888783 [ 31633 ] {} <Debug> ConfigProcessor: Processing configuration file '/etc/clickhouse-server/executable_dictionary.xml'.
2026.05.06 08:48:35.888893 [ 32423 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:35 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE728EE5C985B
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:35.889198 [ 32423 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:35.889272 [ 32395 ] {} <Debug> DatabaseAtomic (test_xfltqef0): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.889318 [ 32395 ] {} <Debug> DatabaseAtomic (test_xfltqef0): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.889368 [ 32395 ] {} <Test> DatabaseAtomic (test_xfltqef0): Counting attached table mt to database test_xfltqef0
2026.05.06 08:48:35.889523 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_xfltqef0.mt' with status OK
2026.05.06 08:48:35.889589 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.bug_67742' in BackgroundLoad
2026.05.06 08:48:35.889632 [ 32395 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.bug_67742
2026.05.06 08:48:35.890005 [ 32450 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0/default_compression_codec.txt)
2026.05.06 08:48:35.890006 [ 32344 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0/default_compression_codec.txt)
2026.05.06 08:48:35.890215 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:35.890215 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:35.890443 [ 32381 ] {} <Debug> test_1.parallel_replicas_plain (279f15c9-7375-4751-ab6e-5597ab10808d): Loading data parts
2026.05.06 08:48:35.890462 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:35.890463 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:35.890752 [ 32381 ] {} <Test> test_1.parallel_replicas_plain (279f15c9-7375-4751-ab6e-5597ab10808d): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.890804 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.890815 [ 32381 ] {} <Trace> test_1.parallel_replicas_plain (279f15c9-7375-4751-ab6e-5597ab10808d): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.890804 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.890841 [ 32370 ] {} <Debug> test_1.dst (3d46120d-95fc-4e38-a1a9-a82eb7298a1d): Loading data parts
2026.05.06 08:48:35.890964 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -10, current is 2126
2026.05.06 08:48:35.891056 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: 786f9b3fa20c683a39db0316dde3086f, offset: 0, size: 10
2026.05.06 08:48:35.891114 [ 32370 ] {} <Test> test_1.dst (3d46120d-95fc-4e38-a1a9-a82eb7298a1d): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.891122 [ 32397 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Loaded data parts (3 items)
2026.05.06 08:48:35.891154 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 10, current is 646
2026.05.06 08:48:35.891187 [ 32370 ] {} <Trace> test_1.dst (3d46120d-95fc-4e38-a1a9-a82eb7298a1d): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.891181 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:ActiveParts:32344, kind: Regular, unbound: 0
2026.05.06 08:48:35.891233 [ 32397 ] {} <Information> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.891273 [ 32450 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: 786f9b3fa20c683a39db0316dde3086f, offset: 0, size: 10
2026.05.06 08:48:35.891402 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:ActiveParts:32450, kind: Regular, unbound: 0
2026.05.06 08:48:35.891428 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:35.891476 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:35.891537 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:35.891554 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.891538 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:35.891595 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.891660 [ 32397 ] {} <Test> DatabaseAtomic (test_1): Counting attached table insert_dedup_token2 to database test_1
2026.05.06 08:48:35.891765 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_54.txt entry, commands size: 1
2026.05.06 08:48:35.891790 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.insert_dedup_token2' with status OK
2026.05.06 08:48:35.891777 [ 32344 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:35.891777 [ 32450 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:35.891906 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.data2013' in BackgroundLoad
2026.05.06 08:48:35.891950 [ 32397 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.data2013
2026.05.06 08:48:35.892052 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_11.txt entry, commands size: 1
2026.05.06 08:48:35.892256 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_63.txt entry, commands size: 1
2026.05.06 08:48:35.892293 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.892339 [ 32374 ] {} <Debug> test_3.ordinary_distinct (099b70b3-e6de-4109-a87e-f1f5a4b35dfe): Loading data parts
2026.05.06 08:48:35.892348 [ 32397 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.892365 [ 32344 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.892401 [ 32397 ] {} <Test> DatabaseAtomic (test_1): Counting attached table data2013 to database test_1
2026.05.06 08:48:35.892427 [ 32344 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Finished loading Active part all_1_1_0 on disk s3_disk
2026.05.06 08:48:35.892435 [ 32450 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): loadDataPart: inserting all_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.892479 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_22.txt entry, commands size: 1
2026.05.06 08:48:35.892478 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.data2013' with status OK
2026.05.06 08:48:35.892496 [ 32450 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Finished loading Active part all_0_0_0 on disk s3_disk
2026.05.06 08:48:35.892560 [ 32374 ] {} <Test> test_3.ordinary_distinct (099b70b3-e6de-4109-a87e-f1f5a4b35dfe): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.892607 [ 32374 ] {} <Debug> test_3.ordinary_distinct (099b70b3-e6de-4109-a87e-f1f5a4b35dfe): There are no data parts
2026.05.06 08:48:35.892632 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.small' in BackgroundLoad
2026.05.06 08:48:35.892671 [ 32397 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.small
2026.05.06 08:48:35.892769 [ 32340 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 2 unexpected data parts. They will be loaded asynchronously
2026.05.06 08:48:35.892826 [ 32423 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:35.892942 [ 32340 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Loaded data parts (2 items)
2026.05.06 08:48:35.893009 [ 32374 ] {} <Debug> DatabaseAtomic (test_3): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.893032 [ 32340 ] {} <Information> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.893114 [ 32374 ] {} <Debug> DatabaseAtomic (test_3): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.893192 [ 32390 ] {} <Debug> shard_1.data_02346 (c47e6d88-ab96-4fd3-948b-5fe6c4145aab): Loading data parts
2026.05.06 08:48:35.893203 [ 32423 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:35 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE728EE9B0128
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:35.893224 [ 32395 ] {} <Debug> test_1.bug_67742 (70154974-534f-4dea-90d9-852642a8e4db): Loading data parts
2026.05.06 08:48:35.893199 [ 32374 ] {} <Test> DatabaseAtomic (test_3): Counting attached table ordinary_distinct to database test_3
2026.05.06 08:48:35.893300 [ 32343 ] {} <Test> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.893367 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table test_3.ordinary_distinct' with status OK
2026.05.06 08:48:35.893374 [ 32340 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.893398 [ 32343 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Will load 3 parts using up to 64 threads
2026.05.06 08:48:35.893410 [ 32340 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.893415 [ 32433 ] {} <Trace> test_1.dst (3d46120d-95fc-4e38-a1a9-a82eb7298a1d): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.893435 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.mt_with_pk' in BackgroundLoad
2026.05.06 08:48:35.893455 [ 32340 ] {} <Test> DatabaseAtomic (test_13): Counting attached table rmt1 to database test_13
2026.05.06 08:48:35.893458 [ 32390 ] {} <Test> shard_1.data_02346 (c47e6d88-ab96-4fd3-948b-5fe6c4145aab): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.893499 [ 32395 ] {} <Test> test_1.bug_67742 (70154974-534f-4dea-90d9-852642a8e4db): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.893526 [ 32423 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:35.893532 [ 32390 ] {} <Trace> shard_1.data_02346 (c47e6d88-ab96-4fd3-948b-5fe6c4145aab): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.893552 [ 32395 ] {} <Trace> test_1.bug_67742 (70154974-534f-4dea-90d9-852642a8e4db): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.893552 [ 32445 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Loading Active part all_0_0_0 from disk default
2026.05.06 08:48:35.893590 [ 32374 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.mt_with_pk
2026.05.06 08:48:35.893677 [ 32339 ] {} <Test> test_13.json_columns (050ab0f2-ba47-4659-8212-6898acc8cf75): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.893692 [ 32398 ] {} <Debug> test_13.tab_00625 (7e1286e4-5364-42ab-bc1f-f524ec05cde2): Loading data parts
2026.05.06 08:48:35.893692 [ 32340 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.rmt1' with status OK
2026.05.06 08:48:35.893723 [ 32339 ] {} <Debug> test_13.json_columns (050ab0f2-ba47-4659-8212-6898acc8cf75): There are no data parts
2026.05.06 08:48:35.893706 [ 32436 ] {} <Trace> test_1.bug_67742 (70154974-534f-4dea-90d9-852642a8e4db): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.893773 [ 32340 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.rmt2' in BackgroundLoad
2026.05.06 08:48:35.893875 [ 32340 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.rmt2
2026.05.06 08:48:35.894159 [ 32339 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.894205 [ 32339 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.894258 [ 32339 ] {} <Test> DatabaseAtomic (test_13): Counting attached table json_columns to database test_13
2026.05.06 08:48:35.894380 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.json_columns' with status OK
2026.05.06 08:48:35.894444 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.data_01509' in BackgroundLoad
2026.05.06 08:48:35.894504 [ 32339 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.data_01509
2026.05.06 08:48:35.895013 [ 32445 ] {} <Test> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): loadDataPart: inserting all_0_0_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.895063 [ 32445 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Finished loading Active part all_0_0_0 on disk default
2026.05.06 08:48:35.895288 [ 32401 ] {} <Debug> test_13.agg_func_col (71d1dd24-191b-434a-b861-c33c346269d2): Loading data parts
2026.05.06 08:48:35.895377 [ 32403 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.895438 [ 32403 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Will load 100 parts using up to 64 threads
2026.05.06 08:48:35.892786 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_23.txt entry, commands size: 1
2026.05.06 08:48:35.895614 [ 32436 ] {} <Test> test_1.bug_67742 (70154974-534f-4dea-90d9-852642a8e4db): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.895660 [ 32436 ] {} <Trace> test_1.bug_67742 (70154974-534f-4dea-90d9-852642a8e4db): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.895833 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_34.txt entry, commands size: 1
2026.05.06 08:48:35.895933 [ 32371 ] {} <Test> test_13.t_modify_to_nullable (cf6d4d1c-040f-4e41-9e38-934ae9036995): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.896025 [ 32424 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260524_20_20_0 from disk default
2026.05.06 08:48:35.896106 [ 32395 ] {} <Debug> test_1.bug_67742 (70154974-534f-4dea-90d9-852642a8e4db): Loaded data parts (1 items)
2026.05.06 08:48:35.896148 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_53.txt entry, commands size: 1
2026.05.06 08:48:35.895994 [ 32371 ] {} <Trace> test_13.t_modify_to_nullable (cf6d4d1c-040f-4e41-9e38-934ae9036995): Will load 2 parts using up to 64 threads
2026.05.06 08:48:35.896358 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_17.txt entry, commands size: 1
2026.05.06 08:48:35.896362 [ 32451 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.896545 [ 32418 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.896602 [ 32449 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260725_82_82_0 from disk default
2026.05.06 08:48:35.896632 [ 32444 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260622_49_49_0 from disk default
2026.05.06 08:48:35.896661 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_27.txt entry, commands size: 1
2026.05.06 08:48:35.896884 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_2.txt entry, commands size: 1
2026.05.06 08:48:35.897102 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_18.txt entry, commands size: 1
2026.05.06 08:48:35.897334 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_6.txt entry, commands size: 1
2026.05.06 08:48:35.897555 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_26.txt entry, commands size: 1
2026.05.06 08:48:35.897790 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_31.txt entry, commands size: 1
2026.05.06 08:48:35.897841 [ 32397 ] {} <Debug> test_13.small (5d27aa9f-7483-4673-91ba-a924ffc45887): Loading data parts
2026.05.06 08:48:35.897985 [ 32337 ] {} <Trace> test_13.t_modify_to_nullable (cf6d4d1c-040f-4e41-9e38-934ae9036995): Loading Active part 2_2_2_0 from disk default
2026.05.06 08:48:35.898078 [ 32418 ] {} <Test> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.898102 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_13.txt entry, commands size: 1
2026.05.06 08:48:35.898128 [ 32418 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.898242 [ 32418 ] {} <Trace> test_13.t_modify_to_nullable (cf6d4d1c-040f-4e41-9e38-934ae9036995): Loading Active part 1_1_1_0 from disk default
2026.05.06 08:48:35.898402 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_9.txt entry, commands size: 1
2026.05.06 08:48:35.898650 [ 32424 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260524_20_20_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.898697 [ 32424 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260524_20_20_0 on disk default
2026.05.06 08:48:35.898710 [ 32451 ] {} <Test> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.898730 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_58.txt entry, commands size: 1
2026.05.06 08:48:35.898757 [ 32451 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.898965 [ 32441 ] {} <Trace> test_1.parallel_replicas_plain (279f15c9-7375-4751-ab6e-5597ab10808d): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.899015 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_20.txt entry, commands size: 1
2026.05.06 08:48:35.899225 [ 32339 ] {} <Debug> test_13.data_01509 (5306d916-8cb2-445d-8b10-24122a32e948): Loading data parts
2026.05.06 08:48:35.899523 [ 32340 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/mti/squcthkasvnsuplvzoqaybjdktdne (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/format_version.txt)
2026.05.06 08:48:35.899657 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(98e8bed7d70180dd805707a7b9c12d5f): Reset state for seek to position 0
2026.05.06 08:48:35.899700 [ 32435 ] {} <Trace> shard_1.data_02346 (c47e6d88-ab96-4fd3-948b-5fe6c4145aab): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.899767 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(98e8bed7d70180dd805707a7b9c12d5f): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:35.899882 [ 32449 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260725_82_82_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.899948 [ 32449 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260725_82_82_0 on disk default
2026.05.06 08:48:35.899956 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(98e8bed7d70180dd805707a7b9c12d5f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: 98e8bed7d70180dd805707a7b9c12d5f, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:BackgroundLoad:32340, kind: Regular, unbound: 0
2026.05.06 08:48:35.900041 [ 32340 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -1, current is 2116
2026.05.06 08:48:35.900094 [ 31633 ] {} <Debug> ConfigProcessor: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/executable_dictionary.xml'.
2026.05.06 08:48:35.900115 [ 32340 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: 98e8bed7d70180dd805707a7b9c12d5f, offset: 0, size: 1
2026.05.06 08:48:35.900173 [ 32340 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 1, current is 656
2026.05.06 08:48:35.900235 [ 32340 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: 98e8bed7d70180dd805707a7b9c12d5f, offset: 0, size: 1
2026.05.06 08:48:35.900345 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(98e8bed7d70180dd805707a7b9c12d5f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: 98e8bed7d70180dd805707a7b9c12d5f, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:BackgroundLoad:32340, kind: Regular, unbound: 0
2026.05.06 08:48:35.900368 [ 32444 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260622_49_49_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.900417 [ 32444 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260622_49_49_0 on disk default
2026.05.06 08:48:35.900431 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(98e8bed7d70180dd805707a7b9c12d5f): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:35.900494 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(98e8bed7d70180dd805707a7b9c12d5f): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:35.900608 [ 32340 ] {} <Test> CachedOnDiskReadBufferFromFile(98e8bed7d70180dd805707a7b9c12d5f): Key: 98e8bed7d70180dd805707a7b9c12d5f. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:35.900787 [ 32337 ] {} <Test> test_13.t_modify_to_nullable (cf6d4d1c-040f-4e41-9e38-934ae9036995): loadDataPart: inserting 2_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.900810 [ 32401 ] {} <Test> test_13.agg_func_col (71d1dd24-191b-434a-b861-c33c346269d2): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.900832 [ 32337 ] {} <Trace> test_13.t_modify_to_nullable (cf6d4d1c-040f-4e41-9e38-934ae9036995): Finished loading Active part 2_2_2_0 on disk default
2026.05.06 08:48:35.900869 [ 32401 ] {} <Trace> test_13.agg_func_col (71d1dd24-191b-434a-b861-c33c346269d2): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.901002 [ 32425 ] {} <Trace> test_13.agg_func_col (71d1dd24-191b-434a-b861-c33c346269d2): Loading Active part 19700101_19700101_1_1_0 from disk default
2026.05.06 08:48:35.901088 [ 32397 ] {} <Test> test_13.small (5d27aa9f-7483-4673-91ba-a924ffc45887): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.901143 [ 32397 ] {} <Trace> test_13.small (5d27aa9f-7483-4673-91ba-a924ffc45887): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.901210 [ 32418 ] {} <Test> test_13.t_modify_to_nullable (cf6d4d1c-040f-4e41-9e38-934ae9036995): loadDataPart: inserting 1_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.901236 [ 32433 ] {} <Test> test_1.dst (3d46120d-95fc-4e38-a1a9-a82eb7298a1d): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.901266 [ 32388 ] {} <Trace> test_13.small (5d27aa9f-7483-4673-91ba-a924ffc45887): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.901264 [ 32441 ] {} <Test> test_1.parallel_replicas_plain (279f15c9-7375-4751-ab6e-5597ab10808d): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.901411 [ 32441 ] {} <Trace> test_1.parallel_replicas_plain (279f15c9-7375-4751-ab6e-5597ab10808d): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.901490 [ 32435 ] {} <Test> shard_1.data_02346 (c47e6d88-ab96-4fd3-948b-5fe6c4145aab): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.901269 [ 32418 ] {} <Trace> test_13.t_modify_to_nullable (cf6d4d1c-040f-4e41-9e38-934ae9036995): Finished loading Active part 1_1_1_0 on disk default
2026.05.06 08:48:35.901567 [ 32381 ] {} <Debug> test_1.parallel_replicas_plain (279f15c9-7375-4751-ab6e-5597ab10808d): Loaded data parts (1 items)
2026.05.06 08:48:35.901592 [ 32453 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260617_44_44_0 from disk default
2026.05.06 08:48:35.901604 [ 32351 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260523_19_19_0 from disk default
2026.05.06 08:48:35.901622 [ 32419 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260602_29_29_0 from disk default
2026.05.06 08:48:35.901630 [ 32343 ] {} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Loaded data parts (3 items)
2026.05.06 08:48:35.901547 [ 32435 ] {} <Trace> shard_1.data_02346 (c47e6d88-ab96-4fd3-948b-5fe6c4145aab): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.896553 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.901625 [ 32432 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260718_75_75_0 from disk default
2026.05.06 08:48:35.901774 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.901791 [ 32428 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260801_89_89_0 from disk default
2026.05.06 08:48:35.901823 [ 32371 ] {} <Debug> test_13.t_modify_to_nullable (cf6d4d1c-040f-4e41-9e38-934ae9036995): Loaded data parts (2 items)
2026.05.06 08:48:35.901727 [ 32343 ] {} <Information> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.901834 [ 32390 ] {} <Debug> shard_1.data_02346 (c47e6d88-ab96-4fd3-948b-5fe6c4145aab): Loaded data parts (1 items)
2026.05.06 08:48:35.901886 [ 32395 ] {} <Test> DatabaseAtomic (test_1): Counting attached table bug_67742 to database test_1
2026.05.06 08:48:35.901916 [ 32398 ] {} <Test> test_13.tab_00625 (7e1286e4-5364-42ab-bc1f-f524ec05cde2): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.901978 [ 32398 ] {} <Trace> test_13.tab_00625 (7e1286e4-5364-42ab-bc1f-f524ec05cde2): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.902052 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.bug_67742' with status OK
2026.05.06 08:48:35.902097 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.902119 [ 32377 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260707_64_64_0 from disk default
2026.05.06 08:48:35.902128 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.dest_view_tiny_log' in BackgroundLoad
2026.05.06 08:48:35.902149 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.902183 [ 32395 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.dest_view_tiny_log
2026.05.06 08:48:35.902218 [ 32381 ] {} <Test> DatabaseAtomic (test_1): Counting attached table parallel_replicas_plain to database test_1
2026.05.06 08:48:35.902250 [ 31633 ] {} <Trace> ExternalDictionariesLoader: Loading config file '/etc/clickhouse-server/executable_pool_dictionary.xml'.
2026.05.06 08:48:35.902327 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.902331 [ 32390 ] {} <Debug> DatabaseAtomic (shard_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.902354 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.parallel_replicas_plain' with status OK
2026.05.06 08:48:35.902393 [ 32390 ] {} <Debug> DatabaseAtomic (shard_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.902437 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table 01053_db.table_for_dict' in BackgroundLoad
2026.05.06 08:48:35.902453 [ 32390 ] {} <Test> DatabaseAtomic (shard_1): Counting attached table data_02346 to database shard_1
2026.05.06 08:48:35.902502 [ 32381 ] {} <Trace> DatabaseAtomic (01053_db): Loading table 01053_db.table_for_dict
2026.05.06 08:48:35.902602 [ 32390 ] {} <Debug> AsyncLoader: Finish load job 'load table shard_1.data_02346' with status OK
2026.05.06 08:48:35.902709 [ 32395 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.902750 [ 32395 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.901777 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_56.txt entry, commands size: 1
2026.05.06 08:48:35.902801 [ 32395 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table dest_view_tiny_log to database test_truncate_database
2026.05.06 08:48:35.902998 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.903032 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_43.txt entry, commands size: 1
2026.05.06 08:48:35.903045 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.903100 [ 32371 ] {} <Test> DatabaseAtomic (test_13): Counting attached table t_modify_to_nullable to database test_13
2026.05.06 08:48:35.902393 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.903182 [ 32343 ] {} <Test> DatabaseAtomic (test_1): Counting attached table insert_dedup_token1 to database test_1
2026.05.06 08:48:35.903248 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_37.txt entry, commands size: 1
2026.05.06 08:48:35.903452 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_19.txt entry, commands size: 1
2026.05.06 08:48:35.902750 [ 32390 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.concurrent_alter_column' in BackgroundLoad
2026.05.06 08:48:35.903689 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_46.txt entry, commands size: 1
2026.05.06 08:48:35.903826 [ 32390 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.concurrent_alter_column
2026.05.06 08:48:35.903833 [ 32425 ] {} <Test> test_13.agg_func_col (71d1dd24-191b-434a-b861-c33c346269d2): loadDataPart: inserting 19700101_19700101_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.903894 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.dest_view_tiny_log' with status OK
2026.05.06 08:48:35.903898 [ 32425 ] {} <Trace> test_13.agg_func_col (71d1dd24-191b-434a-b861-c33c346269d2): Finished loading Active part 19700101_19700101_1_1_0 on disk default
2026.05.06 08:48:35.903958 [ 32419 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260602_29_29_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.904017 [ 32419 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260602_29_29_0 on disk default
2026.05.06 08:48:35.904036 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_39.txt entry, commands size: 1
2026.05.06 08:48:35.903966 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_3.unit' in BackgroundLoad
2026.05.06 08:48:35.904019 [ 32339 ] {} <Test> test_13.data_01509 (5306d916-8cb2-445d-8b10-24122a32e948): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.904066 [ 32401 ] {} <Debug> test_13.agg_func_col (71d1dd24-191b-434a-b861-c33c346269d2): Loaded data parts (1 items)
2026.05.06 08:48:35.904125 [ 32395 ] {} <Trace> DatabaseAtomic (test_3): Loading table test_3.unit
2026.05.06 08:48:35.904140 [ 32388 ] {} <Test> test_13.small (5d27aa9f-7483-4673-91ba-a924ffc45887): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.904189 [ 32388 ] {} <Trace> test_13.small (5d27aa9f-7483-4673-91ba-a924ffc45887): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.904046 [ 32432 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260718_75_75_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.904225 [ 32342 ] {} <Trace> test_13.tab_00625 (7e1286e4-5364-42ab-bc1f-f524ec05cde2): Loading Active part 20260506_20260506_1_2_1 from disk default
2026.05.06 08:48:35.904256 [ 32387 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260505_1_1_0 from disk default
2026.05.06 08:48:35.904299 [ 32432 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260718_75_75_0 on disk default
2026.05.06 08:48:35.904345 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_59.txt entry, commands size: 1
2026.05.06 08:48:35.904367 [ 32376 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260724_81_81_0 from disk default
2026.05.06 08:48:35.904476 [ 32395 ] {} <Debug> DatabaseAtomic (test_3): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.904529 [ 32395 ] {} <Debug> DatabaseAtomic (test_3): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.904572 [ 32395 ] {} <Test> DatabaseAtomic (test_3): Counting attached table unit to database test_3
2026.05.06 08:48:35.904608 [ 32433 ] {} <Trace> test_1.dst (3d46120d-95fc-4e38-a1a9-a82eb7298a1d): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.904612 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_3.txt entry, commands size: 1
2026.05.06 08:48:35.904668 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_3.unit' with status OK
2026.05.06 08:48:35.904757 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.dynamic_test_1' in BackgroundLoad
2026.05.06 08:48:35.904802 [ 32395 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.dynamic_test_1
2026.05.06 08:48:35.904813 [ 32370 ] {} <Debug> test_1.dst (3d46120d-95fc-4e38-a1a9-a82eb7298a1d): Loaded data parts (1 items)
2026.05.06 08:48:35.904920 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_51.txt entry, commands size: 1
2026.05.06 08:48:35.905117 [ 32401 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.905160 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_41.txt entry, commands size: 1
2026.05.06 08:48:35.905188 [ 32401 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.905232 [ 32340 ] {} <Debug> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Loading data parts
2026.05.06 08:48:35.905266 [ 32401 ] {} <Test> DatabaseAtomic (test_13): Counting attached table agg_func_col to database test_13
2026.05.06 08:48:35.905129 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.905432 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk 03008_s3_plain_rewritable eligible for search: false (mode ANY)
2026.05.06 08:48:35.905451 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.905458 [ 32401 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.agg_func_col' with status OK
2026.05.06 08:48:35.905504 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_32.txt entry, commands size: 1
2026.05.06 08:48:35.905544 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk azure eligible for search: true (mode ANY)
2026.05.06 08:48:35.905585 [ 32401 ] {} <Debug> AsyncLoader: Execute load job 'load table test_3.distinct_in_order' in BackgroundLoad
2026.05.06 08:48:35.905608 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk backups eligible for search: true (mode ANY)
2026.05.06 08:48:35.905667 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk azure eligible for search: true (mode ANY)
2026.05.06 08:48:35.905673 [ 32401 ] {} <Trace> DatabaseAtomic (test_3): Loading table test_3.distinct_in_order
2026.05.06 08:48:35.905703 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk default eligible for search: true (mode ANY)
2026.05.06 08:48:35.905784 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_8.txt entry, commands size: 1
2026.05.06 08:48:35.905790 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk disk1_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.905930 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk disk2_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.905992 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk disk3_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.906037 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk disk4_02961 eligible for search: true (mode ANY)
2026.05.06 08:48:35.906079 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_24.txt entry, commands size: 1
2026.05.06 08:48:35.906086 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.906157 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.906199 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk local_disk_3 eligible for search: true (mode ANY)
2026.05.06 08:48:35.906234 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk local_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.906287 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk local_disk_2 eligible for search: true (mode ANY)
2026.05.06 08:48:35.906320 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_61.txt entry, commands size: 1
2026.05.06 08:48:35.906401 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk local_disk_3 eligible for search: true (mode ANY)
2026.05.06 08:48:35.906456 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.906500 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.906536 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.906575 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.906610 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.906624 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_45.txt entry, commands size: 1
2026.05.06 08:48:35.906642 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk s3_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.906678 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk s3_disk_02944 eligible for search: true (mode ANY)
2026.05.06 08:48:35.906693 [ 32339 ] {} <Trace> test_13.data_01509 (5306d916-8cb2-445d-8b10-24122a32e948): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.906788 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk s3_disk_02963 eligible for search: true (mode ANY)
2026.05.06 08:48:35.906840 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk s3_no_cache eligible for search: true (mode ANY)
2026.05.06 08:48:35.906840 [ 32423 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/dklbspircelrvhncgiwwqcgkngqcfwkp/columns.txt (store/fef/fef9e575-f89d-43f4-933c-47fb46bf5de1/0_1_1_1/columns.txt)
2026.05.06 08:48:35.906852 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_64.txt entry, commands size: 1
2026.05.06 08:48:35.906883 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): is disk s3_plain_disk eligible for search: true (mode ANY)
2026.05.06 08:48:35.906907 [ 32446 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260530_26_26_0 from disk default
2026.05.06 08:48:35.904306 [ 32453 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260617_44_44_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.907044 [ 32453 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260617_44_44_0 on disk default
2026.05.06 08:48:35.907071 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_7.txt entry, commands size: 1
2026.05.06 08:48:35.907118 [ 32428 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260801_89_89_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.907157 [ 32397 ] {} <Debug> test_13.small (5d27aa9f-7483-4673-91ba-a924ffc45887): Loaded data parts (1 items)
2026.05.06 08:48:35.907191 [ 32428 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260801_89_89_0 on disk default
2026.05.06 08:48:35.907214 [ 32377 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260707_64_64_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.907295 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_25.txt entry, commands size: 1
2026.05.06 08:48:35.907298 [ 32377 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260707_64_64_0 on disk default
2026.05.06 08:48:35.907314 [ 32351 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260523_19_19_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.907375 [ 32351 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260523_19_19_0 on disk default
2026.05.06 08:48:35.907377 [ 32387 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260505_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.907441 [ 32387 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260505_1_1_0 on disk default
2026.05.06 08:48:35.907446 [ 32376 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260724_81_81_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.907523 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_33.txt entry, commands size: 1
2026.05.06 08:48:35.907530 [ 32376 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260724_81_81_0 on disk default
2026.05.06 08:48:35.907531 [ 32395 ] {} <Test> DatabaseAtomic (test_1): Counting attached table dynamic_test_1 to database test_1
2026.05.06 08:48:35.907691 [ 32370 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.907733 [ 32370 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.907740 [ 32343 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.insert_dedup_token1' with status OK
2026.05.06 08:48:35.907753 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_4.txt entry, commands size: 1
2026.05.06 08:48:35.907771 [ 32370 ] {} <Test> DatabaseAtomic (test_1): Counting attached table dst to database test_1
2026.05.06 08:48:35.907851 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.907872 [ 32343 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.fact_cpc_clicks' in BackgroundLoad
2026.05.06 08:48:35.907902 [ 32397 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.907918 [ 32343 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.fact_cpc_clicks
2026.05.06 08:48:35.907954 [ 32397 ] {} <Test> DatabaseAtomic (test_13): Counting attached table small to database test_13
2026.05.06 08:48:35.908043 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.t_modify_to_nullable' with status OK
2026.05.06 08:48:35.908064 [ 32402 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading mutation: mutation_28.txt entry, commands size: 1
2026.05.06 08:48:35.908122 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_3.parsing_with_names' in BackgroundLoad
2026.05.06 08:48:35.908175 [ 32371 ] {} <Trace> DatabaseAtomic (test_3): Loading table test_3.parsing_with_names
2026.05.06 08:48:35.908188 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.908284 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.dynamic_test_1' with status OK
2026.05.06 08:48:35.908303 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.908349 [ 32343 ] {} <Test> DatabaseAtomic (test_1): Counting attached table fact_cpc_clicks to database test_1
2026.05.06 08:48:35.908398 [ 32402 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.908434 [ 32402 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.908478 [ 32402 ] {} <Test> DatabaseAtomic (test_15): Counting attached table mt_00763_1 to database test_15
2026.05.06 08:48:35.908690 [ 32378 ] {} <Trace> test_13.data_01509 (5306d916-8cb2-445d-8b10-24122a32e948): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.908870 [ 32374 ] {} <Debug> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Loading data parts
2026.05.06 08:48:35.908979 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.03811_sparse_column_aggregation_with_sum' in BackgroundLoad
2026.05.06 08:48:35.909034 [ 32395 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.03811_sparse_column_aggregation_with_sum
2026.05.06 08:48:35.909131 [ 32437 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260629_56_56_0 from disk default
2026.05.06 08:48:35.909198 [ 32446 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260530_26_26_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.909252 [ 32446 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260530_26_26_0 on disk default
2026.05.06 08:48:35.910912 [ 32378 ] {} <Test> test_13.data_01509 (5306d916-8cb2-445d-8b10-24122a32e948): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.910967 [ 32378 ] {} <Trace> test_13.data_01509 (5306d916-8cb2-445d-8b10-24122a32e948): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.911073 [ 32448 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260709_66_66_0 from disk default
2026.05.06 08:48:35.911080 [ 32421 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260731_88_88_0 from disk default
2026.05.06 08:48:35.911159 [ 32344 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260728_85_85_0 from disk default
2026.05.06 08:48:35.911723 [ 32365 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260630_57_57_0 from disk default
2026.05.06 08:48:35.911750 [ 32374 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.911774 [ 32450 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260705_62_62_0 from disk default
2026.05.06 08:48:35.911816 [ 32374 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.911828 [ 32339 ] {} <Debug> test_13.data_01509 (5306d916-8cb2-445d-8b10-24122a32e948): Loaded data parts (1 items)
2026.05.06 08:48:35.911985 [ 32431 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Loading Active part 201810_1_4_2_3 from disk default
2026.05.06 08:48:35.912177 [ 32340 ] {} <Test> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.912180 [ 32401 ] {} <Debug> test_3.distinct_in_order (4892299c-2463-41f8-85d8-cb3ac23d86b9): Loading data parts
2026.05.06 08:48:35.912246 [ 32340 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Will load 2 parts using up to 64 threads
2026.05.06 08:48:35.912344 [ 32339 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.912396 [ 32339 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.912421 [ 32436 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Loading Active part all_0_0_0_2 from disk s3_disk
2026.05.06 08:48:35.912451 [ 32339 ] {} <Test> DatabaseAtomic (test_13): Counting attached table data_01509 to database test_13
2026.05.06 08:48:35.912476 [ 32401 ] {} <Test> test_3.distinct_in_order (4892299c-2463-41f8-85d8-cb3ac23d86b9): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.912497 [ 32445 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Loading Active part all_1_1_0_2 from disk s3_disk
2026.05.06 08:48:35.912539 [ 32401 ] {} <Debug> test_3.distinct_in_order (4892299c-2463-41f8-85d8-cb3ac23d86b9): There are no data parts
2026.05.06 08:48:35.912625 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.data_01509' with status OK
2026.05.06 08:48:35.912730 [ 32339 ] {} <Debug> AsyncLoader: Execute load job 'load table test.hits_dst' in BackgroundLoad
2026.05.06 08:48:35.912796 [ 32339 ] {} <Trace> DatabaseAtomic (test): Loading table test.hits_dst
2026.05.06 08:48:35.912822 [ 32395 ] {} <Debug> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Loading data parts
2026.05.06 08:48:35.912981 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.dst' with status OK
2026.05.06 08:48:35.913009 [ 32401 ] {} <Debug> DatabaseAtomic (test_3): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.913043 [ 32436 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_0_0_0_2/columns.txt)
2026.05.06 08:48:35.913060 [ 32401 ] {} <Debug> DatabaseAtomic (test_3): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.913077 [ 32445 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_1_1_0_2/columns.txt)
2026.05.06 08:48:35.913048 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table shard_0.to' in BackgroundLoad
2026.05.06 08:48:35.913108 [ 32438 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260508_4_4_0 from disk default
2026.05.06 08:48:35.913127 [ 32401 ] {} <Test> DatabaseAtomic (test_3): Counting attached table distinct_in_order to database test_3
2026.05.06 08:48:35.913132 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:35.913136 [ 32381 ] {} <Debug> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Loading data parts
2026.05.06 08:48:35.913154 [ 32370 ] {} <Trace> DatabaseAtomic (shard_0): Loading table shard_0.to
2026.05.06 08:48:35.913167 [ 32442 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260609_36_36_0 from disk default
2026.05.06 08:48:35.913189 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:35.913207 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:35.913239 [ 32422 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260511_7_7_0 from disk default
2026.05.06 08:48:35.913260 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.small' with status OK
2026.05.06 08:48:35.913275 [ 32451 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260702_59_59_0 from disk default
2026.05.06 08:48:35.913306 [ 32449 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260619_46_46_0 from disk default
2026.05.06 08:48:35.913264 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:35.913343 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.913388 [ 32337 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260531_27_27_0 from disk default
2026.05.06 08:48:35.913398 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -67, current is 2115
2026.05.06 08:48:35.913451 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.913457 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: c19724c2f984163665bde94656daf920, offset: 0, size: 67
2026.05.06 08:48:35.913475 [ 32434 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260722_79_79_0 from disk default
2026.05.06 08:48:35.913532 [ 32418 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260811_99_99_0 from disk default
2026.05.06 08:48:35.913554 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.913604 [ 32426 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260506_2_2_0 from disk default
2026.05.06 08:48:35.913625 [ 32435 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260517_13_13_0 from disk default
2026.05.06 08:48:35.913632 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:35.913699 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:35.913764 [ 32419 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260526_22_22_0 from disk default
2026.05.06 08:48:35.913792 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:35.913795 [ 32453 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260510_6_6_0 from disk default
2026.05.06 08:48:35.913922 [ 32424 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260509_5_5_0 from disk default
2026.05.06 08:48:35.913987 [ 32428 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260729_86_86_0 from disk default
2026.05.06 08:48:35.914002 [ 32425 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260522_18_18_0 from disk default
2026.05.06 08:48:35.913405 [ 32395 ] {} <Test> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.914058 [ 32441 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260621_48_48_0 from disk default
2026.05.06 08:48:35.914081 [ 32395 ] {} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Will load 4 parts using up to 64 threads
2026.05.06 08:48:35.914117 [ 32443 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260706_63_63_0 from disk default
2026.05.06 08:48:35.914144 [ 32376 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260806_94_94_0 from disk default
2026.05.06 08:48:35.914157 [ 32430 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260615_42_42_0 from disk default
2026.05.06 08:48:35.913540 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 67, current is 657
2026.05.06 08:48:35.914236 [ 32378 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260513_9_9_0 from disk default
2026.05.06 08:48:35.914261 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: c19724c2f984163665bde94656daf920, offset: 0, size: 67
2026.05.06 08:48:35.914289 [ 32444 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260716_73_73_0 from disk default
2026.05.06 08:48:35.914342 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.914347 [ 32445 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_1_1_0_2/serialization.json)
2026.05.06 08:48:35.914403 [ 32432 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260723_80_80_0 from disk default
2026.05.06 08:48:35.914442 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:35.914457 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:35.914500 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:35.914547 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:35.914583 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:35.914666 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.914701 [ 32397 ] {} <Debug> AsyncLoader: Execute load job 'load table test.visits' in BackgroundLoad
2026.05.06 08:48:35.914712 [ 32381 ] {} <Test> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.914752 [ 32397 ] {} <Trace> DatabaseAtomic (test): Loading table test.visits
2026.05.06 08:48:35.914757 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.914776 [ 32381 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.914794 [ 31633 ] {} <Debug> ConfigProcessor: Processing configuration file '/etc/clickhouse-server/executable_pool_dictionary.xml'.
2026.05.06 08:48:35.914819 [ 32343 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.fact_cpc_clicks' with status OK
2026.05.06 08:48:35.914864 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:35.914876 [ 32343 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.seq' in BackgroundLoad
2026.05.06 08:48:35.914945 [ 32343 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.seq
2026.05.06 08:48:35.914944 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:35.915000 [ 32436 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_0_0_0_2/serialization.json)
2026.05.06 08:48:35.915031 [ 32402 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.mt_00763_1' with status OK
2026.05.06 08:48:35.915054 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:35.915065 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:35.915088 [ 32402 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.t_missed_subcolumns' in BackgroundLoad
2026.05.06 08:48:35.915107 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:35.915166 [ 32402 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.t_missed_subcolumns
2026.05.06 08:48:35.915212 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.915255 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -205, current is 2048
2026.05.06 08:48:35.915280 [ 32439 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260515_11_11_0 from disk default
2026.05.06 08:48:35.915290 [ 32401 ] {} <Debug> AsyncLoader: Finish load job 'load table test_3.distinct_in_order' with status OK
2026.05.06 08:48:35.915301 [ 32351 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260521_17_17_0 from disk default
2026.05.06 08:48:35.915293 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: 138075dc4f908b8696832089aba07a75, offset: 0, size: 205
2026.05.06 08:48:35.915367 [ 32377 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260603_30_30_0 from disk default
2026.05.06 08:48:35.915372 [ 32388 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260605_32_32_0 from disk default
2026.05.06 08:48:35.915393 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 205, current is 724
2026.05.06 08:48:35.915410 [ 32401 ] {} <Debug> AsyncLoader: Execute load job 'load table test.hits_s3' in BackgroundLoad
2026.05.06 08:48:35.915444 [ 32433 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260721_78_78_0 from disk default
2026.05.06 08:48:35.915457 [ 32387 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260611_38_38_0 from disk default
2026.05.06 08:48:35.915624 [ 32445 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:35.916194 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:35.916258 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:35.916275 [ 32429 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260613_40_40_0 from disk default
2026.05.06 08:48:35.916378 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.916401 [ 32440 ] {} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Loading Active part all_1_40_3 from disk default
2026.05.06 08:48:35.916465 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.916546 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:35.916585 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:35.916666 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:35.916016 [ 32446 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260715_72_72_0 from disk default
2026.05.06 08:48:35.916100 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: 138075dc4f908b8696832089aba07a75, offset: 0, size: 205
2026.05.06 08:48:35.916080 [ 32454 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Loading Active part all_1_4_1 from disk default
2026.05.06 08:48:35.916805 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.916887 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:35.916932 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:35.917023 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:35.917151 [ 32445 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_1_1_0_2/checksums.txt)
2026.05.06 08:48:35.917211 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:35.917275 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:35.917380 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.917429 [ 32445 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -276, current is 1843
2026.05.06 08:48:35.917470 [ 32445 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: c07538018b3df63ed790dfe1d5992011, offset: 0, size: 276
2026.05.06 08:48:35.917523 [ 32445 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 276, current is 929
2026.05.06 08:48:35.917563 [ 32445 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: c07538018b3df63ed790dfe1d5992011, offset: 0, size: 276
2026.05.06 08:48:35.917611 [ 32436 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:35.917638 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.917672 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:35.917708 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:35.917737 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:35.917749 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:35.917848 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:35.917854 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.917949 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.918018 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:35.918056 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:35.918135 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:35.918363 [ 32445 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:35.918419 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:35.918476 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:35.918593 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.918615 [ 32436 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_0_0_0_2/checksums.txt)
2026.05.06 08:48:35.918671 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:35.918675 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.918732 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:35.918740 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:35.918780 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:35.918845 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.918877 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:35.918894 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -276, current is 1567
2026.05.06 08:48:35.918954 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: 996dedb7dc3c5a127eefbc86122b0cfc, offset: 0, size: 276
2026.05.06 08:48:35.918990 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 276, current is 1205
2026.05.06 08:48:35.919165 [ 32436 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: 996dedb7dc3c5a127eefbc86122b0cfc, offset: 0, size: 276
2026.05.06 08:48:35.919248 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.919325 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:35.919366 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:35.919448 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:35.919512 [ 32445 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_1_1_0_2/count.txt)
2026.05.06 08:48:35.919586 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:35.919671 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:35.919777 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.919880 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.919954 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:35.919993 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:35.920012 [ 32436 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:35.920073 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:35.920077 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:35.920157 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:35.920267 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.920349 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.920415 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:35.920453 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:35.920541 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:35.921062 [ 32436 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_0_0_0_2/count.txt)
2026.05.06 08:48:35.921073 [ 32445 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:35.921117 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:35.921131 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:35.921178 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:35.921189 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:35.921284 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.921295 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.921386 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.921399 [ 32445 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -10, current is 1291
2026.05.06 08:48:35.921441 [ 32445 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: 1f29f94719cc1b8336e4b3a854c89221, offset: 0, size: 10
2026.05.06 08:48:35.921448 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:35.921496 [ 32445 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 10, current is 1481
2026.05.06 08:48:35.921500 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:35.921536 [ 32445 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: 1f29f94719cc1b8336e4b3a854c89221, offset: 0, size: 10
2026.05.06 08:48:35.921582 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:35.921584 [ 32401 ] {} <Trace> DatabaseAtomic (test): Loading table test.hits_s3
2026.05.06 08:48:35.921611 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:ActiveParts:32445, kind: Regular, unbound: 0
2026.05.06 08:48:35.921682 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:35.921723 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:35.921809 [ 32445 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:35.922177 [ 32445 ] {} <Test> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): loadDataPart: inserting all_1_1_0_2 (state Active) into data_parts_indexes
2026.05.06 08:48:35.922238 [ 32445 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Finished loading Active part all_1_1_0_2 on disk s3_disk
2026.05.06 08:48:35.922557 [ 32436 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/8ef/8ef9e61f-45fe-4d95-8c98-3a5c8ab69768/all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:35.922614 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:35.922668 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:35.922770 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.922835 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:ActiveParts:32436, kind: Regular, unbound: 0
2026.05.06 08:48:35.922884 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:35.922912 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:35.922974 [ 32436 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:35.923075 [ 32445 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260812_100_100_0 from disk default
2026.05.06 08:48:35.923187 [ 32436 ] {} <Test> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): loadDataPart: inserting all_0_0_0_2 (state Active) into data_parts_indexes
2026.05.06 08:48:35.923224 [ 32436 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Finished loading Active part all_0_0_0_2 on disk s3_disk
2026.05.06 08:48:35.923340 [ 32436 ] {} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Loading Active part all_47_47_0 from disk default
2026.05.06 08:48:35.923342 [ 32340 ] {} <Debug> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Loaded data parts (2 items)
2026.05.06 08:48:35.923461 [ 32455 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260620_47_47_0 from disk default
2026.05.06 08:48:35.924101 [ 32456 ] {} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Loading Active part all_41_46_1 from disk default
2026.05.06 08:48:35.924345 [ 32343 ] {} <Debug> test_1.seq (c77107c0-3482-422e-a7b4-a6ca5925f17c): Loading data parts
2026.05.06 08:48:35.924114 [ 32457 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260803_91_91_0 from disk default
2026.05.06 08:48:35.924693 [ 32458 ] {} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Loading Active part all_48_48_0 from disk default
2026.05.06 08:48:35.924785 [ 32340 ] {} <Information> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.925199 [ 32340 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.925254 [ 32340 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.924751 [ 32440 ] {} <Test> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): loadDataPart: inserting all_1_40_3 (state Active) into data_parts_indexes
2026.05.06 08:48:35.925314 [ 32340 ] {} <Test> DatabaseAtomic (test_13): Counting attached table rmt2 to database test_13
2026.05.06 08:48:35.925341 [ 32440 ] {} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Finished loading Active part all_1_40_3 on disk default
2026.05.06 08:48:35.925344 [ 32436 ] {} <Test> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): loadDataPart: inserting all_47_47_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.924775 [ 32460 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260703_60_60_0 from disk default
2026.05.06 08:48:35.925412 [ 32436 ] {} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Finished loading Active part all_47_47_0 on disk default
2026.05.06 08:48:35.925492 [ 32340 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.rmt2' with status OK
2026.05.06 08:48:35.925568 [ 32436 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260710_67_67_0 from disk default
2026.05.06 08:48:35.925573 [ 32340 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.tab' in BackgroundLoad
2026.05.06 08:48:35.925628 [ 32340 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.tab
2026.05.06 08:48:35.925754 [ 32454 ] {} <Test> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): loadDataPart: inserting all_1_4_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.925753 [ 32343 ] {} <Test> test_1.seq (c77107c0-3482-422e-a7b4-a6ca5925f17c): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.925815 [ 32454 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Finished loading Active part all_1_4_1 on disk default
2026.05.06 08:48:35.925874 [ 32343 ] {} <Trace> test_1.seq (c77107c0-3482-422e-a7b4-a6ca5925f17c): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.925948 [ 32381 ] {} <Debug> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Found 4 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:35.924889 [ 32459 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260726_83_83_0 from disk default
2026.05.06 08:48:35.926015 [ 32381 ] {} <Debug> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Loaded data parts (1 items)
2026.05.06 08:48:35.926311 [ 32440 ] {} <Trace> test_1.seq (c77107c0-3482-422e-a7b4-a6ca5925f17c): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.926368 [ 32402 ] {} <Debug> test_1.t_missed_subcolumns (8818baf7-a966-4838-9649-2360598a4bb3): Loading data parts
2026.05.06 08:48:35.926679 [ 32454 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260520_16_16_0 from disk default
2026.05.06 08:48:35.926777 [ 31633 ] {} <Debug> ConfigProcessor: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/executable_pool_dictionary.xml'.
2026.05.06 08:48:35.927034 [ 32381 ] {} <Debug> DatabaseAtomic (01053_db): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.927072 [ 32381 ] {} <Debug> DatabaseAtomic (01053_db): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.927112 [ 32381 ] {} <Test> DatabaseAtomic (01053_db): Counting attached table table_for_dict to database 01053_db
2026.05.06 08:48:35.927239 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table 01053_db.table_for_dict' with status OK
2026.05.06 08:48:35.927299 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test.basic' in BackgroundLoad
2026.05.06 08:48:35.927340 [ 32381 ] {} <Trace> DatabaseAtomic (test): Loading table test.basic
2026.05.06 08:48:35.928094 [ 32370 ] {} <Debug> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Loading data parts
2026.05.06 08:48:35.928351 [ 31633 ] {} <Trace> ExternalDictionariesLoader: Loading config file '/etc/clickhouse-server/ints_dictionary.xml'.
2026.05.06 08:48:35.927274 [ 32461 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260623_50_50_0 from disk default
2026.05.06 08:48:35.928377 [ 32464 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260527_23_23_0 from disk default
2026.05.06 08:48:35.928547 [ 32440 ] {} <Test> test_1.seq (c77107c0-3482-422e-a7b4-a6ca5925f17c): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.928598 [ 32440 ] {} <Trace> test_1.seq (c77107c0-3482-422e-a7b4-a6ca5925f17c): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.928704 [ 32440 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260719_76_76_0 from disk default
2026.05.06 08:48:35.928714 [ 32343 ] {} <Debug> test_1.seq (c77107c0-3482-422e-a7b4-a6ca5925f17c): Loaded data parts (1 items)
2026.05.06 08:48:35.929099 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.929150 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.929210 [ 32343 ] {} <Test> DatabaseAtomic (test_1): Counting attached table seq to database test_1
2026.05.06 08:48:35.929328 [ 32343 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.seq' with status OK
2026.05.06 08:48:35.929538 [ 32402 ] {} <Test> test_1.t_missed_subcolumns (8818baf7-a966-4838-9649-2360598a4bb3): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.929596 [ 32342 ] {} <Test> test_13.tab_00625 (7e1286e4-5364-42ab-bc1f-f524ec05cde2): loadDataPart: inserting 20260506_20260506_1_2_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.929606 [ 32402 ] {} <Trace> test_1.t_missed_subcolumns (8818baf7-a966-4838-9649-2360598a4bb3): Will load 2 parts using up to 64 threads
2026.05.06 08:48:35.929661 [ 32342 ] {} <Trace> test_13.tab_00625 (7e1286e4-5364-42ab-bc1f-f524ec05cde2): Finished loading Active part 20260506_20260506_1_2_1 on disk default
2026.05.06 08:48:35.929859 [ 32343 ] {} <Debug> AsyncLoader: Execute load job 'load table test.hits_buffer' in BackgroundLoad
2026.05.06 08:48:35.930011 [ 32343 ] {} <Trace> DatabaseAtomic (test): Loading table test.hits_buffer
2026.05.06 08:48:35.930105 [ 32462 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260805_93_93_0 from disk default
2026.05.06 08:48:35.930360 [ 32398 ] {} <Debug> test_13.tab_00625 (7e1286e4-5364-42ab-bc1f-f524ec05cde2): Loaded data parts (1 items)
2026.05.06 08:48:35.930379 [ 32381 ] {} <Debug> DatabaseAtomic (test): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.930391 [ 32342 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260701_58_58_0 from disk default
2026.05.06 08:48:35.930416 [ 32381 ] {} <Debug> DatabaseAtomic (test): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.930459 [ 32381 ] {} <Test> DatabaseAtomic (test): Counting attached table basic to database test
2026.05.06 08:48:35.930529 [ 32458 ] {} <Test> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): loadDataPart: inserting all_48_48_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.930597 [ 32458 ] {} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Finished loading Active part all_48_48_0 on disk default
2026.05.06 08:48:35.930685 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test.basic' with status OK
2026.05.06 08:48:35.930963 [ 32398 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.930616 [ 32456 ] {} <Test> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): loadDataPart: inserting all_41_46_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.930973 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.src' in BackgroundLoad
2026.05.06 08:48:35.931013 [ 32398 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.931034 [ 32381 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.src
2026.05.06 08:48:35.931044 [ 32456 ] {} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Finished loading Active part all_41_46_1 on disk default
2026.05.06 08:48:35.931056 [ 32398 ] {} <Test> DatabaseAtomic (test_13): Counting attached table tab_00625 to database test_13
2026.05.06 08:48:35.931102 [ 32458 ] {} <Trace> test_1.t_missed_subcolumns (8818baf7-a966-4838-9649-2360598a4bb3): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.931170 [ 32395 ] {} <Debug> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Loaded data parts (4 items)
2026.05.06 08:48:35.931181 [ 32398 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.tab_00625' with status OK
2026.05.06 08:48:35.931257 [ 32398 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.override_test__fuzz_45' in BackgroundLoad
2026.05.06 08:48:35.931332 [ 32398 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.override_test__fuzz_45
2026.05.06 08:48:35.931666 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.931687 [ 32370 ] {} <Test> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.931720 [ 32456 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260808_96_96_0 from disk default
2026.05.06 08:48:35.931746 [ 32370 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.931800 [ 32340 ] {} <Debug> test_1.tab (336dcd17-9a32-4d7a-83fb-ef98f3c42f2c): Loading data parts
2026.05.06 08:48:35.931165 [ 32463 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260610_37_37_0 from disk default
2026.05.06 08:48:35.931958 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.932003 [ 32381 ] {} <Test> DatabaseAtomic (test_1): Counting attached table src to database test_1
2026.05.06 08:48:35.932075 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.932112 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.932154 [ 32395 ] {} <Test> DatabaseAtomic (test_1): Counting attached table 03811_sparse_column_aggregation_with_sum to database test_1
2026.05.06 08:48:35.931927 [ 32466 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260804_92_92_0 from disk default
2026.05.06 08:48:35.932270 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.03811_sparse_column_aggregation_with_sum' with status OK
2026.05.06 08:48:35.932232 [ 32468 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260604_31_31_0 from disk default
2026.05.06 08:48:35.932021 [ 32465 ] {} <Trace> test_1.t_missed_subcolumns (8818baf7-a966-4838-9649-2360598a4bb3): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:35.932735 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.source_table_log' in BackgroundLoad
2026.05.06 08:48:35.932815 [ 32395 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.source_table_log
2026.05.06 08:48:35.932899 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.src' with status OK
2026.05.06 08:48:35.932966 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table shard_0.from_1' in BackgroundLoad
2026.05.06 08:48:35.933011 [ 32381 ] {} <Trace> DatabaseAtomic (shard_0): Loading table shard_0.from_1
2026.05.06 08:48:35.933129 [ 32423 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/dklbspircelrvhncgiwwqcgkngqcfwkp/serialization.json (store/fef/fef9e575-f89d-43f4-933c-47fb46bf5de1/0_1_1_1/serialization.json)
2026.05.06 08:48:35.933196 [ 32467 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260612_39_39_0 from disk default
2026.05.06 08:48:35.933385 [ 32395 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.933758 [ 32395 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.933225 [ 32469 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260802_90_90_0 from disk default
2026.05.06 08:48:35.933828 [ 32395 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table source_table_log to database test_truncate_database
2026.05.06 08:48:35.933929 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.source_table_log' with status OK
2026.05.06 08:48:35.933982 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.test' in BackgroundLoad
2026.05.06 08:48:35.934017 [ 32395 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.test
2026.05.06 08:48:35.933813 [ 32471 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260810_98_98_0 from disk default
2026.05.06 08:48:35.934221 [ 32340 ] {} <Test> test_1.tab (336dcd17-9a32-4d7a-83fb-ef98f3c42f2c): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.934282 [ 32340 ] {} <Trace> test_1.tab (336dcd17-9a32-4d7a-83fb-ef98f3c42f2c): Will load 1 parts using up to 64 threads
2026.05.06 08:48:35.934146 [ 32472 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260624_51_51_0 from disk default
2026.05.06 08:48:35.934785 [ 32458 ] {} <Test> test_1.t_missed_subcolumns (8818baf7-a966-4838-9649-2360598a4bb3): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.934847 [ 32458 ] {} <Trace> test_1.t_missed_subcolumns (8818baf7-a966-4838-9649-2360598a4bb3): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.935046 [ 32458 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260529_25_25_0 from disk default
2026.05.06 08:48:35.934866 [ 32470 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Loading Active part all_0_3_1 from disk default
2026.05.06 08:48:35.935575 [ 32473 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260607_34_34_0 from disk default
2026.05.06 08:48:35.937062 [ 32398 ] {} <Debug> test_13.override_test__fuzz_45 (43d589de-0c5d-4fd9-94bd-d6cb0e6c70a7): Loading data parts
2026.05.06 08:48:35.936860 [ 32475 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260618_45_45_0 from disk default
2026.05.06 08:48:35.937186 [ 32474 ] {} <Trace> test_1.tab (336dcd17-9a32-4d7a-83fb-ef98f3c42f2c): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:35.939287 [ 32465 ] {} <Test> test_1.t_missed_subcolumns (8818baf7-a966-4838-9649-2360598a4bb3): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.939360 [ 32465 ] {} <Trace> test_1.t_missed_subcolumns (8818baf7-a966-4838-9649-2360598a4bb3): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:35.939525 [ 32402 ] {} <Debug> test_1.t_missed_subcolumns (8818baf7-a966-4838-9649-2360598a4bb3): Loaded data parts (2 items)
2026.05.06 08:48:35.939531 [ 32470 ] {} <Test> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): loadDataPart: inserting all_0_3_1 (state Active) into data_parts_indexes
2026.05.06 08:48:35.939634 [ 32470 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Finished loading Active part all_0_3_1 on disk default
2026.05.06 08:48:35.939641 [ 32476 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260601_28_28_0 from disk default
2026.05.06 08:48:35.939749 [ 32370 ] {} <Debug> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Loaded data parts (1 items)
2026.05.06 08:48:35.939765 [ 32470 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260625_52_52_0 from disk default
2026.05.06 08:48:35.939859 [ 32370 ] {} <Information> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.940011 [ 32402 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.940070 [ 32402 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.940137 [ 32402 ] {} <Test> DatabaseAtomic (test_1): Counting attached table t_missed_subcolumns to database test_1
2026.05.06 08:48:35.940199 [ 32370 ] {} <Debug> DatabaseAtomic (shard_0): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.940235 [ 32370 ] {} <Debug> DatabaseAtomic (shard_0): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.940276 [ 32370 ] {} <Test> DatabaseAtomic (shard_0): Counting attached table to to database shard_0
2026.05.06 08:48:35.940283 [ 32402 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.t_missed_subcolumns' with status OK
2026.05.06 08:48:35.940359 [ 32402 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.null' in BackgroundLoad
2026.05.06 08:48:35.940432 [ 32402 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.null
2026.05.06 08:48:35.940532 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table shard_0.to' with status OK
2026.05.06 08:48:35.940619 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.input_function_table_3' in BackgroundLoad
2026.05.06 08:48:35.940658 [ 32370 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.input_function_table_3
2026.05.06 08:48:35.940829 [ 32402 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.940863 [ 32402 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.940903 [ 32402 ] {} <Test> DatabaseAtomic (test_1): Counting attached table null to database test_1
2026.05.06 08:48:35.940973 [ 32402 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.null' with status OK
2026.05.06 08:48:35.941020 [ 32402 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.simple_key_simple_attributes_source_table' in BackgroundLoad
2026.05.06 08:48:35.941038 [ 32370 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.941055 [ 32402 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.simple_key_simple_attributes_source_table
2026.05.06 08:48:35.941072 [ 32370 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.941141 [ 32370 ] {} <Test> DatabaseAtomic (test_13): Counting attached table input_function_table_3 to database test_13
2026.05.06 08:48:35.941223 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.input_function_table_3' with status OK
2026.05.06 08:48:35.941298 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_02771.t' in BackgroundLoad
2026.05.06 08:48:35.941330 [ 32370 ] {} <Trace> DatabaseAtomic (test_02771): Loading table test_02771.t
2026.05.06 08:48:35.941254 [ 32474 ] {} <Test> test_1.tab (336dcd17-9a32-4d7a-83fb-ef98f3c42f2c): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:35.941520 [ 32381 ] {} <Debug> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Loading data parts
2026.05.06 08:48:35.941525 [ 32474 ] {} <Trace> test_1.tab (336dcd17-9a32-4d7a-83fb-ef98f3c42f2c): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:35.941656 [ 32340 ] {} <Debug> test_1.tab (336dcd17-9a32-4d7a-83fb-ef98f3c42f2c): Loaded data parts (1 items)
2026.05.06 08:48:35.941713 [ 32402 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.941756 [ 32402 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.941810 [ 32402 ] {} <Test> DatabaseAtomic (test_13): Counting attached table simple_key_simple_attributes_source_table to database test_13
2026.05.06 08:48:35.941912 [ 32402 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.simple_key_simple_attributes_source_table' with status OK
2026.05.06 08:48:35.941974 [ 32402 ] {} <Debug> AsyncLoader: Execute load job 'load table test.hits' in BackgroundLoad
2026.05.06 08:48:35.942076 [ 32402 ] {} <Trace> DatabaseAtomic (test): Loading table test.hits
2026.05.06 08:48:35.942276 [ 32340 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.942338 [ 32340 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.942438 [ 32340 ] {} <Test> DatabaseAtomic (test_1): Counting attached table tab to database test_1
2026.05.06 08:48:35.942617 [ 32340 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.tab' with status OK
2026.05.06 08:48:35.942711 [ 32340 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.A_M' in BackgroundLoad
2026.05.06 08:48:35.942773 [ 32340 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.A_M
2026.05.06 08:48:35.943573 [ 32381 ] {} <Test> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:35.943648 [ 32381 ] {} <Debug> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): There are no data parts
2026.05.06 08:48:35.943718 [ 32381 ] {} <Information> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:35.943713 [ 32477 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260713_70_70_0 from disk default
2026.05.06 08:48:35.944220 [ 32381 ] {} <Debug> DatabaseAtomic (shard_0): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.944321 [ 32381 ] {} <Debug> DatabaseAtomic (shard_0): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.944383 [ 32381 ] {} <Test> DatabaseAtomic (shard_0): Counting attached table from_1 to database shard_0
2026.05.06 08:48:35.944550 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table shard_0.from_1' with status OK
2026.05.06 08:48:35.944627 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.temp' in BackgroundLoad
2026.05.06 08:48:35.944679 [ 32381 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.temp
2026.05.06 08:48:35.944856 [ 32395 ] {} <Debug> test_1.test (4336dd04-2a15-4ed1-ae01-b52dc1032e5b): Loading data parts
2026.05.06 08:48:35.945194 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:35.945233 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:35.945276 [ 32381 ] {} <Test> DatabaseAtomic (test_1): Counting attached table temp to database test_1
2026.05.06 08:48:35.945375 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.temp' with status OK
2026.05.06 08:48:35.945430 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.test_grouping_sets_predicate' in BackgroundLoad
2026.05.06 08:48:35.945466 [ 32381 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.test_grouping_sets_predicate
2026.05.06 08:48:35.946737 [ 31633 ] {} <Debug> ConfigProcessor: Processing configuration file '/etc/clickhouse-server/ints_dictionary.xml'.
2026.05.06 08:48:35.947264 [ 32370 ] {} <Debug> test_02771.t (21e5e4b4-a867-499f-9011-f1ca3cb51ff0): Loading data parts
2026.05.06 08:48:35.947558 [ 32423 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/dklbspircelrvhncgiwwqcgkngqcfwkp/metadata_version.txt (store/fef/fef9e575-f89d-43f4-933c-47fb46bf5de1/0_1_1_1/metadata_version.txt)
2026.05.06 08:48:35.951053 [ 32381 ] {} <Debug> test_1.test_grouping_sets_predicate (b1e21e55-e983-44cc-935e-c5fba1fe7ac7): Loading data parts
2026.05.06 08:48:35.958073 [ 32423 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/dklbspircelrvhncgiwwqcgkngqcfwkp/checksums.txt (store/fef/fef9e575-f89d-43f4-933c-47fb46bf5de1/0_1_1_1/checksums.txt)
2026.05.06 08:48:35.971975 [ 31633 ] {} <Debug> ConfigProcessor: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/ints_dictionary.xml'.
2026.05.06 08:48:35.973352 [ 31633 ] {} <Trace> ExternalDictionariesLoader: Loading config file '/etc/clickhouse-server/strings_dictionary.xml'.
2026.05.06 08:48:35.978925 [ 32423 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/dklbspircelrvhncgiwwqcgkngqcfwkp/a.cmrk2 (store/fef/fef9e575-f89d-43f4-933c-47fb46bf5de1/0_1_1_1/a.cmrk2)
2026.05.06 08:48:35.981948 [ 31633 ] {} <Debug> ConfigProcessor: Processing configuration file '/etc/clickhouse-server/strings_dictionary.xml'.
2026.05.06 08:48:35.990147 [ 32423 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/dklbspircelrvhncgiwwqcgkngqcfwkp/count.txt (store/fef/fef9e575-f89d-43f4-933c-47fb46bf5de1/0_1_1_1/count.txt)
2026.05.06 08:48:35.995189 [ 31633 ] {} <Debug> ConfigProcessor: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/strings_dictionary.xml'.
2026.05.06 08:48:35.998897 [ 32423 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/dklbspircelrvhncgiwwqcgkngqcfwkp/partition.dat (store/fef/fef9e575-f89d-43f4-933c-47fb46bf5de1/0_1_1_1/partition.dat)
2026.05.06 08:48:36.000011 [ 32368 ] {} <Trace> ExternalDictionariesLoader: Loading config file '71161200-93e8-40fd-b11d-983febc99551'.
2026.05.06 08:48:36.000023 [ 32371 ] {} <Debug> DatabaseAtomic (test_3): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.000145 [ 32371 ] {} <Debug> DatabaseAtomic (test_3): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.000363 [ 32371 ] {} <Test> DatabaseAtomic (test_3): Counting attached table parsing_with_names to database test_3
2026.05.06 08:48:36.000538 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_3.parsing_with_names' with status OK
2026.05.06 08:48:36.000503 [ 31633 ] {} <Trace> ExternalUserDefinedExecutableFunctionsLoader: Loading config file '/etc/clickhouse-server/test_function.xml'.
2026.05.06 08:48:36.000634 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.storage' in BackgroundLoad
2026.05.06 08:48:36.000758 [ 32371 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.storage
2026.05.06 08:48:36.000815 [ 32447 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 2.09 GiB, peak 2.09 GiB, free memory in arenas 0.00 B, will set to 2.22 GiB (RSS), difference: 143.15 MiB
2026.05.06 08:48:36.000975 [ 32368 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.001059 [ 32368 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.001140 [ 32368 ] {} <Test> DatabaseAtomic (test_13): Counting attached table dict_02916_dictionary_access_test_13 to database test_13
2026.05.06 08:48:36.001227 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.001279 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.001494 [ 32368 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.dict_02916_dictionary_access_test_13' with status OK
2026.05.06 08:48:36.001614 [ 32371 ] {} <Test> DatabaseAtomic (test_13): Counting attached table storage to database test_13
2026.05.06 08:48:36.002475 [ 32365 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260630_57_57_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.002565 [ 32365 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260630_57_57_0 on disk default
2026.05.06 08:48:36.002570 [ 32448 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260709_66_66_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.002640 [ 32448 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260709_66_66_0 on disk default
2026.05.06 08:48:36.003047 [ 32344 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260728_85_85_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.003132 [ 32437 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260629_56_56_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.003134 [ 32344 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260728_85_85_0 on disk default
2026.05.06 08:48:36.003192 [ 32437 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260629_56_56_0 on disk default
2026.05.06 08:48:36.003712 [ 32337 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260531_27_27_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.003809 [ 32337 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260531_27_27_0 on disk default
2026.05.06 08:48:36.004037 [ 32435 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260517_13_13_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.004090 [ 32435 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260517_13_13_0 on disk default
2026.05.06 08:48:36.004099 [ 32426 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260506_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.004147 [ 32426 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260506_2_2_0 on disk default
2026.05.06 08:48:36.004311 [ 32453 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260510_6_6_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.004362 [ 32453 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260510_6_6_0 on disk default
2026.05.06 08:48:36.004653 [ 32450 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260705_62_62_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.004797 [ 32450 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260705_62_62_0 on disk default
2026.05.06 08:48:36.005395 [ 32418 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260811_99_99_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.005506 [ 32418 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260811_99_99_0 on disk default
2026.05.06 08:48:36.005748 [ 32368 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.test2' in BackgroundLoad
2026.05.06 08:48:36.005834 [ 32368 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.test2
2026.05.06 08:48:36.005942 [ 32344 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260614_41_41_0 from disk default
2026.05.06 08:48:36.006001 [ 32442 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260609_36_36_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.006192 [ 32449 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260619_46_46_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.006216 [ 32442 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260609_36_36_0 on disk default
2026.05.06 08:48:36.006348 [ 32449 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260619_46_46_0 on disk default
2026.05.06 08:48:36.006778 [ 32434 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260722_79_79_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.006838 [ 32434 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260722_79_79_0 on disk default
2026.05.06 08:48:36.006013 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.storage' with status OK
2026.05.06 08:48:36.007059 [ 32419 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260526_22_22_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.007134 [ 32419 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260526_22_22_0 on disk default
2026.05.06 08:48:36.007142 [ 32422 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260511_7_7_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.007194 [ 32422 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260511_7_7_0 on disk default
2026.05.06 08:48:36.008025 [ 32432 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260723_80_80_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.008083 [ 32432 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260723_80_80_0 on disk default
2026.05.06 08:48:36.008094 [ 32421 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260731_88_88_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.008153 [ 32421 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260731_88_88_0 on disk default
2026.05.06 08:48:36.008176 [ 32438 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260508_4_4_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.008281 [ 32428 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260729_86_86_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.008288 [ 32438 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260508_4_4_0 on disk default
2026.05.06 08:48:36.008340 [ 32428 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260729_86_86_0 on disk default
2026.05.06 08:48:36.008361 [ 32441 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260621_48_48_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.008436 [ 32430 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260615_42_42_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.008435 [ 32441 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260621_48_48_0 on disk default
2026.05.06 08:48:36.008509 [ 32430 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260615_42_42_0 on disk default
2026.05.06 08:48:36.008524 [ 32425 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260522_18_18_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.008585 [ 32425 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260522_18_18_0 on disk default
2026.05.06 08:48:36.008695 [ 32378 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260513_9_9_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.008743 [ 32378 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260513_9_9_0 on disk default
2026.05.06 08:48:36.009137 [ 32433 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260721_78_78_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.009193 [ 32433 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260721_78_78_0 on disk default
2026.05.06 08:48:36.009412 [ 32444 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260716_73_73_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.009522 [ 32444 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260716_73_73_0 on disk default
2026.05.06 08:48:36.010372 [ 32429 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260613_40_40_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.010420 [ 32429 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260613_40_40_0 on disk default
2026.05.06 08:48:36.010466 [ 32376 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260806_94_94_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.010542 [ 32376 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260806_94_94_0 on disk default
2026.05.06 08:48:36.010630 [ 32387 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260611_38_38_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.010711 [ 32387 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260611_38_38_0 on disk default
2026.05.06 08:48:36.010743 [ 32388 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260605_32_32_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.010812 [ 32388 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260605_32_32_0 on disk default
2026.05.06 08:48:36.010817 [ 32446 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260715_72_72_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.010883 [ 32446 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260715_72_72_0 on disk default
2026.05.06 08:48:36.010960 [ 32377 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260603_30_30_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.011026 [ 32377 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260603_30_30_0 on disk default
2026.05.06 08:48:36.011292 [ 32439 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260515_11_11_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.011431 [ 32439 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260515_11_11_0 on disk default
2026.05.06 08:48:36.011458 [ 32351 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260521_17_17_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.011694 [ 32351 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260521_17_17_0 on disk default
2026.05.06 08:48:36.012493 [ 32443 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260706_63_63_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.012546 [ 32443 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260706_63_63_0 on disk default
2026.05.06 08:48:36.013600 [ 32436 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260710_67_67_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.013679 [ 32436 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260710_67_67_0 on disk default
2026.05.06 08:48:36.013707 [ 32445 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260812_100_100_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.013857 [ 32445 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260812_100_100_0 on disk default
2026.05.06 08:48:36.014636 [ 32451 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260702_59_59_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.014685 [ 32451 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260702_59_59_0 on disk default
2026.05.06 08:48:36.014748 [ 32424 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260509_5_5_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.014875 [ 32424 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260509_5_5_0 on disk default
2026.05.06 08:48:36.015045 [ 32458 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260529_25_25_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.015107 [ 32458 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260529_25_25_0 on disk default
2026.05.06 08:48:36.015154 [ 32440 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260719_76_76_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.015238 [ 32440 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260719_76_76_0 on disk default
2026.05.06 08:48:36.015249 [ 32470 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260625_52_52_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.015314 [ 32470 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260625_52_52_0 on disk default
2026.05.06 08:48:36.016023 [ 32343 ] {} <Debug> DatabaseAtomic (test): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.016094 [ 32343 ] {} <Debug> DatabaseAtomic (test): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.016170 [ 32343 ] {} <Test> DatabaseAtomic (test): Counting attached table hits_buffer to database test
2026.05.06 08:48:36.016768 [ 32342 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260701_58_58_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.016862 [ 32342 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260701_58_58_0 on disk default
2026.05.06 08:48:36.018235 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.simple_key_source_table_01862' in BackgroundLoad
2026.05.06 08:48:36.020547 [ 32454 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260520_16_16_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.020666 [ 32454 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260520_16_16_0 on disk default
2026.05.06 08:48:36.020942 [ 32381 ] {} <Test> test_1.test_grouping_sets_predicate (b1e21e55-e983-44cc-935e-c5fba1fe7ac7): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.021008 [ 32381 ] {} <Debug> test_1.test_grouping_sets_predicate (b1e21e55-e983-44cc-935e-c5fba1fe7ac7): There are no data parts
2026.05.06 08:48:36.021149 [ 32435 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260712_69_69_0 from disk default
2026.05.06 08:48:36.021233 [ 32398 ] {} <Test> test_13.override_test__fuzz_45 (43d589de-0c5d-4fd9-94bd-d6cb0e6c70a7): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.021312 [ 32398 ] {} <Trace> test_13.override_test__fuzz_45 (43d589de-0c5d-4fd9-94bd-d6cb0e6c70a7): Will load 1 parts using up to 64 threads
2026.05.06 08:48:36.021380 [ 32395 ] {} <Test> test_1.test (4336dd04-2a15-4ed1-ae01-b52dc1032e5b): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.022114 [ 32370 ] {} <Test> test_02771.t (21e5e4b4-a867-499f-9011-f1ca3cb51ff0): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.022181 [ 32370 ] {} <Debug> test_02771.t (21e5e4b4-a867-499f-9011-f1ca3cb51ff0): There are no data parts
2026.05.06 08:48:36.022308 [ 32418 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260528_24_24_0 from disk default
2026.05.06 08:48:36.022331 [ 32426 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260608_35_35_0 from disk default
2026.05.06 08:48:36.022371 [ 32344 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260614_41_41_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.022471 [ 32344 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260614_41_41_0 on disk default
2026.05.06 08:48:36.023374 [ 32459 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260726_83_83_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.023428 [ 32459 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260726_83_83_0 on disk default
2026.05.06 08:48:36.023816 [ 32472 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260624_51_51_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.023881 [ 32472 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260624_51_51_0 on disk default
2026.05.06 08:48:36.023941 [ 32395 ] {} <Trace> test_1.test (4336dd04-2a15-4ed1-ae01-b52dc1032e5b): Will load 1 parts using up to 64 threads
2026.05.06 08:48:36.023987 [ 32437 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260519_15_15_0 from disk default
2026.05.06 08:48:36.024147 [ 32365 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260730_87_87_0 from disk default
2026.05.06 08:48:36.024158 [ 32448 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260807_95_95_0 from disk default
2026.05.06 08:48:36.024228 [ 32453 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260507_3_3_0 from disk default
2026.05.06 08:48:36.024238 [ 32371 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.simple_key_source_table_01862
2026.05.06 08:48:36.024310 [ 32351 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260727_84_84_0 from disk default
2026.05.06 08:48:36.024586 [ 32466 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260804_92_92_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.024616 [ 32443 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260627_54_54_0 from disk default
2026.05.06 08:48:36.024639 [ 32451 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260512_8_8_0 from disk default
2026.05.06 08:48:36.024700 [ 32477 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260713_70_70_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.024724 [ 32458 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260714_71_71_0 from disk default
2026.05.06 08:48:36.024853 [ 32467 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260612_39_39_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.024954 [ 32467 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260612_39_39_0 on disk default
2026.05.06 08:48:36.025128 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.025216 [ 32431 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): loadDataPart: inserting 201810_1_4_2_3 (state Active) into data_parts_indexes
2026.05.06 08:48:36.025234 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.025270 [ 32467 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260717_74_74_0 from disk default
2026.05.06 08:48:36.025299 [ 32431 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Finished loading Active part 201810_1_4_2_3 on disk default
2026.05.06 08:48:36.025360 [ 32371 ] {} <Test> DatabaseAtomic (test_13): Counting attached table simple_key_source_table_01862 to database test_13
2026.05.06 08:48:36.025530 [ 32374 ] {} <Debug> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Found 5 outdated data parts. They will be loaded asynchronously
2026.05.06 08:48:36.025658 [ 32454 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260516_12_12_0 from disk default
2026.05.06 08:48:36.025662 [ 32374 ] {} <Debug> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Loaded data parts (1 items)
2026.05.06 08:48:36.026164 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.026288 [ 32381 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.026445 [ 32381 ] {} <Test> DatabaseAtomic (test_1): Counting attached table test_grouping_sets_predicate to database test_1
2026.05.06 08:48:36.026768 [ 32374 ] {} <Debug> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Loading mutation: mutation_3.txt entry, commands size: 1
2026.05.06 08:48:36.026842 [ 32465 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260711_68_68_0 from disk default
2026.05.06 08:48:36.026946 [ 32470 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260628_55_55_0 from disk default
2026.05.06 08:48:36.026965 [ 32440 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260708_65_65_0 from disk default
2026.05.06 08:48:36.027053 [ 32442 ] {} <Trace> test_13.override_test__fuzz_45 (43d589de-0c5d-4fd9-94bd-d6cb0e6c70a7): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:36.027464 [ 32477 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260713_70_70_0 on disk default
2026.05.06 08:48:36.027585 [ 32343 ] {} <Debug> AsyncLoader: Finish load job 'load table test.hits_buffer' with status OK
2026.05.06 08:48:36.027795 [ 32449 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260626_53_53_0 from disk default
2026.05.06 08:48:36.027839 [ 32434 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260704_61_61_0 from disk default
2026.05.06 08:48:36.027903 [ 32422 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260616_43_43_0 from disk default
2026.05.06 08:48:36.026785 [ 32445 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260720_77_77_0 from disk default
2026.05.06 08:48:36.027903 [ 32419 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260518_14_14_0 from disk default
2026.05.06 08:48:36.027927 [ 32374 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.027960 [ 32432 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260514_10_10_0 from disk default
2026.05.06 08:48:36.024643 [ 32466 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260804_92_92_0 on disk default
2026.05.06 08:48:36.028037 [ 32421 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260525_21_21_0 from disk default
2026.05.06 08:48:36.028131 [ 32431 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260809_97_97_0 from disk default
2026.05.06 08:48:36.028158 [ 32428 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loading Active part 20260606_33_33_0 from disk default
2026.05.06 08:48:36.025552 [ 32475 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260618_45_45_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.028316 [ 32443 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260627_54_54_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.028330 [ 32475 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260618_45_45_0 on disk default
2026.05.06 08:48:36.028380 [ 32443 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260627_54_54_0 on disk default
2026.05.06 08:48:36.028433 [ 32458 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260714_71_71_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.028574 [ 32451 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260512_8_8_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.028579 [ 32458 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260714_71_71_0 on disk default
2026.05.06 08:48:36.028652 [ 32451 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260512_8_8_0 on disk default
2026.05.06 08:48:36.028659 [ 32365 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260730_87_87_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.028730 [ 32365 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260730_87_87_0 on disk default
2026.05.06 08:48:36.028737 [ 32351 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260727_84_84_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.028798 [ 32351 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260727_84_84_0 on disk default
2026.05.06 08:48:36.028807 [ 32448 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260807_95_95_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.028863 [ 32448 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260807_95_95_0 on disk default
2026.05.06 08:48:36.028908 [ 32461 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260623_50_50_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.028975 [ 32461 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260623_50_50_0 on disk default
2026.05.06 08:48:36.029004 [ 32453 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260507_3_3_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.029088 [ 32453 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260507_3_3_0 on disk default
2026.05.06 08:48:36.029098 [ 32440 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260708_65_65_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.029149 [ 32440 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260708_65_65_0 on disk default
2026.05.06 08:48:36.029171 [ 32454 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260516_12_12_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.029221 [ 32454 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260516_12_12_0 on disk default
2026.05.06 08:48:36.029367 [ 32368 ] {} <Debug> test_1.test2 (dfcccf80-ca63-4a40-a31e-fafd5aa387b5): Loading data parts
2026.05.06 08:48:36.029539 [ 32468 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260604_31_31_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.029589 [ 32468 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260604_31_31_0 on disk default
2026.05.06 08:48:36.030304 [ 32456 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260808_96_96_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.030355 [ 32456 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260808_96_96_0 on disk default
2026.05.06 08:48:36.030949 [ 32470 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260628_55_55_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.030998 [ 32470 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260628_55_55_0 on disk default
2026.05.06 08:48:36.031289 [ 32343 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1.tab2' in BackgroundLoad
2026.05.06 08:48:36.031385 [ 32343 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1.tab2
2026.05.06 08:48:36.031473 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.simple_key_source_table_01862' with status OK
2026.05.06 08:48:36.031565 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.02162_test_table' in BackgroundLoad
2026.05.06 08:48:36.031643 [ 32371 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.02162_test_table
2026.05.06 08:48:36.031653 [ 32462 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260805_93_93_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.031733 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.test_grouping_sets_predicate' with status OK
2026.05.06 08:48:36.031775 [ 32462 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260805_93_93_0 on disk default
2026.05.06 08:48:36.032950 [ 32467 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260717_74_74_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.033000 [ 32467 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260717_74_74_0 on disk default
2026.05.06 08:48:36.033244 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.source_table_stripe_log' in BackgroundLoad
2026.05.06 08:48:36.033449 [ 32418 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260528_24_24_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.033523 [ 32418 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260528_24_24_0 on disk default
2026.05.06 08:48:36.033788 [ 32441 ] {} <Trace> test_1.test (4336dd04-2a15-4ed1-ae01-b52dc1032e5b): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:36.034755 [ 32457 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260803_91_91_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.034855 [ 32457 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260803_91_91_0 on disk default
2026.05.06 08:48:36.034917 [ 32464 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260527_23_23_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.034996 [ 32464 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260527_23_23_0 on disk default
2026.05.06 08:48:36.035280 [ 32442 ] {} <Test> test_13.override_test__fuzz_45 (43d589de-0c5d-4fd9-94bd-d6cb0e6c70a7): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.035291 [ 32476 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260601_28_28_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.035325 [ 32442 ] {} <Trace> test_13.override_test__fuzz_45 (43d589de-0c5d-4fd9-94bd-d6cb0e6c70a7): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:36.035339 [ 32476 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260601_28_28_0 on disk default
2026.05.06 08:48:36.035452 [ 32398 ] {} <Debug> test_13.override_test__fuzz_45 (43d589de-0c5d-4fd9-94bd-d6cb0e6c70a7): Loaded data parts (1 items)
2026.05.06 08:48:36.035782 [ 32437 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260519_15_15_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.035833 [ 32437 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260519_15_15_0 on disk default
2026.05.06 08:48:36.036150 [ 32473 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260607_34_34_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.036265 [ 32473 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260607_34_34_0 on disk default
2026.05.06 08:48:36.036406 [ 32460 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260703_60_60_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.036464 [ 32460 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260703_60_60_0 on disk default
2026.05.06 08:48:36.036549 [ 32422 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260616_43_43_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.036669 [ 32422 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260616_43_43_0 on disk default
2026.05.06 08:48:36.036687 [ 32463 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260610_37_37_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.036758 [ 32463 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260610_37_37_0 on disk default
2026.05.06 08:48:36.036825 [ 32445 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260720_77_77_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.036886 [ 32445 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260720_77_77_0 on disk default
2026.05.06 08:48:36.036908 [ 32465 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260711_68_68_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.036969 [ 32465 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260711_68_68_0 on disk default
2026.05.06 08:48:36.036988 [ 32434 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260704_61_61_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.037055 [ 32434 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260704_61_61_0 on disk default
2026.05.06 08:48:36.037064 [ 32374 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.037143 [ 32374 ] {} <Test> DatabaseAtomic (test_13): Counting attached table mt_with_pk to database test_13
2026.05.06 08:48:36.037224 [ 32426 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260608_35_35_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.037350 [ 32426 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260608_35_35_0 on disk default
2026.05.06 08:48:36.037490 [ 32469 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260802_90_90_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.037544 [ 32469 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260802_90_90_0 on disk default
2026.05.06 08:48:36.038322 [ 32435 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260712_69_69_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.038370 [ 32435 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260712_69_69_0 on disk default
2026.05.06 08:48:36.038386 [ 32449 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260626_53_53_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.038463 [ 32449 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260626_53_53_0 on disk default
2026.05.06 08:48:36.038572 [ 32455 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260620_47_47_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.038654 [ 32455 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260620_47_47_0 on disk default
2026.05.06 08:48:36.038849 [ 32471 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260810_98_98_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.038960 [ 32471 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260810_98_98_0 on disk default
2026.05.06 08:48:36.039156 [ 32431 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260809_97_97_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.039214 [ 32431 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260809_97_97_0 on disk default
2026.05.06 08:48:36.039238 [ 32419 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260518_14_14_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.039306 [ 32419 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260518_14_14_0 on disk default
2026.05.06 08:48:36.039447 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.mt_with_pk' with status OK
2026.05.06 08:48:36.039559 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.02183_dictionary_test_table' in BackgroundLoad
2026.05.06 08:48:36.039637 [ 32374 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.02183_dictionary_test_table
2026.05.06 08:48:36.039660 [ 32432 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260514_10_10_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.039734 [ 32432 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260514_10_10_0 on disk default
2026.05.06 08:48:36.039843 [ 32343 ] {} <Debug> test_1.tab2 (254c0c20-e03e-4a8e-9738-a0df7f2f9753): Loading data parts
2026.05.06 08:48:36.039861 [ 32370 ] {} <Debug> DatabaseAtomic (test_02771): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.040005 [ 32370 ] {} <Debug> DatabaseAtomic (test_02771): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.040082 [ 32441 ] {} <Test> test_1.test (4336dd04-2a15-4ed1-ae01-b52dc1032e5b): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.040131 [ 32441 ] {} <Trace> test_1.test (4336dd04-2a15-4ed1-ae01-b52dc1032e5b): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:36.040161 [ 32370 ] {} <Test> DatabaseAtomic (test_02771): Counting attached table t to database test_02771
2026.05.06 08:48:36.040264 [ 32395 ] {} <Debug> test_1.test (4336dd04-2a15-4ed1-ae01-b52dc1032e5b): Loaded data parts (1 items)
2026.05.06 08:48:36.040317 [ 32374 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.040393 [ 32374 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.040476 [ 32374 ] {} <Test> DatabaseAtomic (test_15): Counting attached table 02183_dictionary_test_table to database test_15
2026.05.06 08:48:36.040479 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_02771.t' with status OK
2026.05.06 08:48:36.040575 [ 32381 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.source_table_stripe_log
2026.05.06 08:48:36.040683 [ 32370 ] {} <Debug> AsyncLoader: Execute load job 'load table test_1..inner_id.9907701c-369a-410f-99dc-ca9afd271de1' in BackgroundLoad
2026.05.06 08:48:36.040761 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.040842 [ 32395 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.040893 [ 32395 ] {} <Test> DatabaseAtomic (test_1): Counting attached table test to database test_1
2026.05.06 08:48:36.040898 [ 32370 ] {} <Trace> DatabaseAtomic (test_1): Loading table test_1..inner_id.9907701c-369a-410f-99dc-ca9afd271de1
2026.05.06 08:48:36.040930 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.02183_dictionary_test_table' with status OK
2026.05.06 08:48:36.041018 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.dest_view_log' in BackgroundLoad
2026.05.06 08:48:36.041068 [ 32374 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.dest_view_log
2026.05.06 08:48:36.041192 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.test' with status OK
2026.05.06 08:48:36.041231 [ 32381 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.041256 [ 32395 ] {} <Debug> AsyncLoader: Execute load job 'load table shard_1.to' in BackgroundLoad
2026.05.06 08:48:36.041282 [ 32381 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.041363 [ 32395 ] {} <Trace> DatabaseAtomic (shard_1): Loading table shard_1.to
2026.05.06 08:48:36.041386 [ 32381 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table source_table_stripe_log to database test_truncate_database
2026.05.06 08:48:36.041474 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.source_table_stripe_log' with status OK
2026.05.06 08:48:36.041506 [ 32374 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.041549 [ 32374 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.041550 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_truncate_database.dest_dictionary' in BackgroundLoad
2026.05.06 08:48:36.041599 [ 32374 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table dest_view_log to database test_truncate_database
2026.05.06 08:48:36.041608 [ 32381 ] {} <Trace> DatabaseAtomic (test_truncate_database): Loading table test_truncate_database.dest_dictionary
2026.05.06 08:48:36.041714 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.dest_view_log' with status OK
2026.05.06 08:48:36.041765 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table 01778_db.hierarchy_flat_dictionary' in BackgroundLoad
2026.05.06 08:48:36.041771 [ 32423 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/dklbspircelrvhncgiwwqcgkngqcfwkp/minmax_a.idx (store/fef/fef9e575-f89d-43f4-933c-47fb46bf5de1/0_1_1_1/minmax_a.idx)
2026.05.06 08:48:36.041801 [ 32374 ] {} <Trace> DatabaseAtomic (01778_db): Loading table 01778_db.hierarchy_flat_dictionary
2026.05.06 08:48:36.041917 [ 31633 ] {} <Debug> ConfigProcessor: Processing configuration file '/etc/clickhouse-server/test_function.xml'.
2026.05.06 08:48:36.041973 [ 32421 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260525_21_21_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.042087 [ 32421 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260525_21_21_0 on disk default
2026.05.06 08:48:36.042520 [ 32428 ] {} <Test> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): loadDataPart: inserting 20260606_33_33_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.042635 [ 32428 ] {} <Trace> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Finished loading Active part 20260606_33_33_0 on disk default
2026.05.06 08:48:36.042825 [ 32340 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.042915 [ 32340 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.043013 [ 32340 ] {} <Test> DatabaseAtomic (test_13): Counting attached table A_M to database test_13
2026.05.06 08:48:36.043024 [ 32381 ] {} <Trace> ExternalDictionariesLoader: Loading config file 'e1157c8f-9651-4b58-824a-103691fe26b6'.
2026.05.06 08:48:36.043138 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.043215 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.043233 [ 32340 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.A_M' with status OK
2026.05.06 08:48:36.043291 [ 32374 ] {} <Trace> ExternalDictionariesLoader: Loading config file 'e56ec67b-a9c3-4381-902c-079b6b746314'.
2026.05.06 08:48:36.043349 [ 32371 ] {} <Test> DatabaseAtomic (test_13): Counting attached table 02162_test_table to database test_13
2026.05.06 08:48:36.043432 [ 32381 ] {} <Debug> DatabaseAtomic (test_truncate_database): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.043449 [ 32340 ] {} <Debug> AsyncLoader: Execute load job 'load table 01945.db.test_dictionary' in BackgroundLoad
2026.05.06 08:48:36.043468 [ 32381 ] {} <Debug> DatabaseAtomic (test_truncate_database): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.043554 [ 32381 ] {} <Test> DatabaseAtomic (test_truncate_database): Counting attached table dest_dictionary to database test_truncate_database
2026.05.06 08:48:36.043550 [ 32398 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.043577 [ 32340 ] {} <Trace> DatabaseAtomic (01945.db): Loading table 01945.db.test_dictionary
2026.05.06 08:48:36.043584 [ 32403 ] {} <Debug> test_13.bug_13492 (9ae2671c-8478-40b4-a211-34eac619f3cc): Loaded data parts (100 items)
2026.05.06 08:48:36.043638 [ 32398 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.043698 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.02162_test_table' with status OK
2026.05.06 08:48:36.043646 [ 32374 ] {} <Debug> DatabaseAtomic (01778_db): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.043768 [ 32398 ] {} <Test> DatabaseAtomic (test_13): Counting attached table override_test__fuzz_45 to database test_13
2026.05.06 08:48:36.043846 [ 32374 ] {} <Debug> DatabaseAtomic (01778_db): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.043853 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.dict_01509' in BackgroundLoad
2026.05.06 08:48:36.043892 [ 32374 ] {} <Test> DatabaseAtomic (01778_db): Counting attached table hierarchy_flat_dictionary to database 01778_db
2026.05.06 08:48:36.043926 [ 32371 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.dict_01509
2026.05.06 08:48:36.044028 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table 01778_db.hierarchy_flat_dictionary' with status OK
2026.05.06 08:48:36.044083 [ 32374 ] {} <Debug> AsyncLoader: Execute load job 'load table 01053_db.ssd_dict' in BackgroundLoad
2026.05.06 08:48:36.044125 [ 32374 ] {} <Trace> DatabaseAtomic (01053_db): Loading table 01053_db.ssd_dict
2026.05.06 08:48:36.044241 [ 32398 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.override_test__fuzz_45' with status OK
2026.05.06 08:48:36.044305 [ 32398 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.hashed_array_dictionary_simple_key_simple_attributes' in BackgroundLoad
2026.05.06 08:48:36.044357 [ 32398 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.hashed_array_dictionary_simple_key_simple_attributes
2026.05.06 08:48:36.044356 [ 31633 ] {} <Debug> ConfigProcessor: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/test_function.xml'.
2026.05.06 08:48:36.044445 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_truncate_database.dest_dictionary' with status OK
2026.05.06 08:48:36.044565 [ 32368 ] {} <Test> test_1.test2 (dfcccf80-ca63-4a40-a31e-fafd5aa387b5): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.044645 [ 32368 ] {} <Trace> test_1.test2 (dfcccf80-ca63-4a40-a31e-fafd5aa387b5): Will load 1 parts using up to 64 threads
2026.05.06 08:48:36.044661 [ 32381 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.simple_key_hashed_dictionary_01862' in BackgroundLoad
2026.05.06 08:48:36.044683 [ 32343 ] {} <Test> test_1.tab2 (254c0c20-e03e-4a8e-9738-a0df7f2f9753): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.044735 [ 32343 ] {} <Trace> test_1.tab2 (254c0c20-e03e-4a8e-9738-a0df7f2f9753): Will load 2 parts using up to 64 threads
2026.05.06 08:48:36.044739 [ 32381 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.simple_key_hashed_dictionary_01862
2026.05.06 08:48:36.044805 [ 32425 ] {} <Trace> test_1.test2 (dfcccf80-ca63-4a40-a31e-fafd5aa387b5): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:36.044992 [ 32376 ] {} <Trace> test_1.tab2 (254c0c20-e03e-4a8e-9738-a0df7f2f9753): Loading Active part all_1_1_0 from disk default
2026.05.06 08:48:36.045449 [ 31633 ] {} <Trace> ExternalUserDefinedExecutableFunctionsLoader: Will load 'test_function' because always_load_everything flag is set.
2026.05.06 08:48:36.045671 [ 31633 ] {} <Trace> ExternalUserDefinedExecutableFunctionsLoader: Will load the object 'test_function' immediately, force = false, loading_id = 1
2026.05.06 08:48:36.046003 [ 31633 ] {} <Trace> ExternalUserDefinedExecutableFunctionsLoader: Start loading object 'test_function'
2026.05.06 08:48:36.046056 [ 32371 ] {} <Trace> ExternalDictionariesLoader: Loading config file 'b4c7de75-3d1a-4b04-822d-0177baf9641c'.
2026.05.06 08:48:36.046341 [ 32381 ] {} <Trace> ExternalDictionariesLoader: Loading config file '7cead437-bf5d-4424-8ba0-3977fac8fce1'.
2026.05.06 08:48:36.046585 [ 32403 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.046633 [ 32398 ] {} <Trace> ExternalDictionariesLoader: Loading config file '97f1625b-b420-40a0-ac0e-b1f4e14ed0d9'.
2026.05.06 08:48:36.046687 [ 32403 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.046789 [ 31633 ] {} <Trace> ExternalUserDefinedExecutableFunctionsLoader: Supposed update time for 'test_function' is never (loaded, lifetime 0)
2026.05.06 08:48:36.046801 [ 32403 ] {} <Test> DatabaseAtomic (test_13): Counting attached table bug_13492 to database test_13
2026.05.06 08:48:36.046887 [ 32374 ] {} <Trace> ExternalDictionariesLoader: Loading config file '701b3c4b-fac3-4317-a21a-efc26faff67c'.
2026.05.06 08:48:36.046927 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.046941 [ 31633 ] {} <Trace> ExternalUserDefinedExecutableFunctionsLoader: Next update time for 'test_function' was set to 294247-01-10 05:00:54
2026.05.06 08:48:36.046980 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.047013 [ 32376 ] {} <Test> test_1.tab2 (254c0c20-e03e-4a8e-9738-a0df7f2f9753): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.047044 [ 32403 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.bug_13492' with status OK
2026.05.06 08:48:36.047081 [ 32376 ] {} <Trace> test_1.tab2 (254c0c20-e03e-4a8e-9738-a0df7f2f9753): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:36.047107 [ 32371 ] {} <Test> DatabaseAtomic (test_13): Counting attached table dict_01509 to database test_13
2026.05.06 08:48:36.047117 [ 32403 ] {} <Debug> AsyncLoader: Execute load job 'load table test_15.02183_hashed_array_dictionary' in BackgroundLoad
2026.05.06 08:48:36.047174 [ 32403 ] {} <Trace> DatabaseAtomic (test_15): Loading table test_15.02183_hashed_array_dictionary
2026.05.06 08:48:36.047182 [ 32381 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.047230 [ 32381 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.047281 [ 32381 ] {} <Test> DatabaseAtomic (test_13): Counting attached table simple_key_hashed_dictionary_01862 to database test_13
2026.05.06 08:48:36.047288 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.dict_01509' with status OK
2026.05.06 08:48:36.047294 [ 32374 ] {} <Debug> DatabaseAtomic (01053_db): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.047341 [ 32374 ] {} <Debug> DatabaseAtomic (01053_db): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.047346 [ 32398 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.047359 [ 32371 ] {} <Debug> AsyncLoader: Execute load job 'load table test_13.02162_test_dictionary' in BackgroundLoad
2026.05.06 08:48:36.047450 [ 32374 ] {} <Test> DatabaseAtomic (01053_db): Counting attached table ssd_dict to database 01053_db
2026.05.06 08:48:36.047458 [ 32398 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.047501 [ 32371 ] {} <Trace> DatabaseAtomic (test_13): Loading table test_13.02162_test_dictionary
2026.05.06 08:48:36.047519 [ 32398 ] {} <Test> DatabaseAtomic (test_13): Counting attached table hashed_array_dictionary_simple_key_simple_attributes to database test_13
2026.05.06 08:48:36.047523 [ 32273 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 524601 rows, containing 6 columns (6 merged, 0 gathered) in 2.565445426 sec., 204487.29670229202 rows/sec., 4.76 MiB/sec.
2026.05.06 08:48:36.047709 [ 32429 ] {} <Trace> test_1.tab2 (254c0c20-e03e-4a8e-9738-a0df7f2f9753): Loading Active part all_2_2_0 from disk default
2026.05.06 08:48:36.047746 [ 32374 ] {} <Debug> AsyncLoader: Finish load job 'load table 01053_db.ssd_dict' with status OK
2026.05.06 08:48:36.047809 [ 32374 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.047963 [ 32381 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.simple_key_hashed_dictionary_01862' with status OK
2026.05.06 08:48:36.048012 [ 32381 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.048195 [ 32403 ] {} <Trace> ExternalDictionariesLoader: Loading config file '9000190c-31ea-4742-a77d-a0d20c9cb3fe'.
2026.05.06 08:48:36.048289 [ 32398 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.hashed_array_dictionary_simple_key_simple_attributes' with status OK
2026.05.06 08:48:36.048348 [ 32398 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.048891 [ 31633 ] {} <Debug> AsyncLoader: Schedule load job 'startup ddl worker' into BackgrndStartup
2026.05.06 08:48:36.049177 [ 32395 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Loading data parts
2026.05.06 08:48:36.049227 [ 32340 ] {} <Trace> ExternalDictionariesLoader: Loading config file 'a079a21a-41a1-4345-9b89-606f0b5dd6b8'.
2026.05.06 08:48:36.049387 [ 32403 ] {} <Debug> DatabaseAtomic (test_15): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.049415 [ 32429 ] {} <Test> test_1.tab2 (254c0c20-e03e-4a8e-9738-a0df7f2f9753): loadDataPart: inserting all_2_2_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.049456 [ 32403 ] {} <Debug> DatabaseAtomic (test_15): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.049462 [ 32429 ] {} <Trace> test_1.tab2 (254c0c20-e03e-4a8e-9738-a0df7f2f9753): Finished loading Active part all_2_2_0 on disk default
2026.05.06 08:48:36.049530 [ 32395 ] {} <Test> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.049549 [ 32403 ] {} <Test> DatabaseAtomic (test_15): Counting attached table 02183_hashed_array_dictionary to database test_15
2026.05.06 08:48:36.049588 [ 32395 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): There are no data parts
2026.05.06 08:48:36.049625 [ 31633 ] {} <Information> Application: Listening for http://[::]:8123
2026.05.06 08:48:36.049647 [ 32395 ] {} <Information> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Table will be in readonly mode until initialization is finished
2026.05.06 08:48:36.049689 [ 32403 ] {} <Debug> AsyncLoader: Finish load job 'load table test_15.02183_hashed_array_dictionary' with status OK
2026.05.06 08:48:36.049697 [ 32343 ] {} <Debug> test_1.tab2 (254c0c20-e03e-4a8e-9738-a0df7f2f9753): Loaded data parts (2 items)
2026.05.06 08:48:36.049801 [ 32403 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.049808 [ 32423 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:36.049976 [ 31633 ] {} <Information> Application: Listening for https://[::]:8443
2026.05.06 08:48:36.050016 [ 32370 ] {} <Debug> test_1.`.inner_id.9907701c-369a-410f-99dc-ca9afd271de1` (b774ffc7-db57-4ddf-adc4-f3e95bfde8be): Loading data parts
2026.05.06 08:48:36.050074 [ 32395 ] {} <Debug> DatabaseAtomic (shard_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.050252 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.050262 [ 32395 ] {} <Debug> DatabaseAtomic (shard_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.050455 [ 32343 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.050494 [ 31633 ] {} <Information> Application: Listening for native protocol (tcp): [::]:9000
2026.05.06 08:48:36.050515 [ 32343 ] {} <Test> DatabaseAtomic (test_1): Counting attached table tab2 to database test_1
2026.05.06 08:48:36.050471 [ 32370 ] {} <Test> test_1.`.inner_id.9907701c-369a-410f-99dc-ca9afd271de1` (b774ffc7-db57-4ddf-adc4-f3e95bfde8be): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.050598 [ 32395 ] {} <Test> DatabaseAtomic (shard_1): Counting attached table to to database shard_1
2026.05.06 08:48:36.050628 [ 32343 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.tab2' with status OK
2026.05.06 08:48:36.050621 [ 32370 ] {} <Debug> test_1.`.inner_id.9907701c-369a-410f-99dc-ca9afd271de1` (b774ffc7-db57-4ddf-adc4-f3e95bfde8be): There are no data parts
2026.05.06 08:48:36.050673 [ 32343 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.050678 [ 32423 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:36 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE728F7F681A3
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:36.050719 [ 31633 ] {} <Information> Application: Listening for native protocol (tcp) with PROXY: [::]:9010
2026.05.06 08:48:36.050745 [ 32340 ] {} <Debug> DatabaseAtomic (01945.db): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.050820 [ 31633 ] {} <Information> Application: Listening for secure native protocol (tcp_secure): [::]:9440
2026.05.06 08:48:36.050842 [ 32340 ] {} <Debug> DatabaseAtomic (01945.db): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.050859 [ 32395 ] {} <Debug> AsyncLoader: Finish load job 'load table shard_1.to' with status OK
2026.05.06 08:48:36.050928 [ 32340 ] {} <Test> DatabaseAtomic (01945.db): Counting attached table test_dictionary to database 01945.db
2026.05.06 08:48:36.050943 [ 32395 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.051133 [ 32340 ] {} <Debug> AsyncLoader: Finish load job 'load table 01945.db.test_dictionary' with status OK
2026.05.06 08:48:36.051180 [ 32340 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.051584 [ 32370 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.051643 [ 31633 ] {} <Information> Application: Listening for MySQL compatibility protocol: [::]:9004
2026.05.06 08:48:36.051704 [ 32370 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.051809 [ 32370 ] {} <Test> DatabaseAtomic (test_1): Counting attached table .inner_id.9907701c-369a-410f-99dc-ca9afd271de1 to database test_1
2026.05.06 08:48:36.051890 [ 32371 ] {} <Trace> ExternalDictionariesLoader: Loading config file '9a4dceb0-e264-42cf-a806-eebbe724dabd'.
2026.05.06 08:48:36.051898 [ 31633 ] {} <Information> Application: Listening for PostgreSQL compatibility protocol: [::]:9005
2026.05.06 08:48:36.052041 [ 32370 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1..inner_id.9907701c-369a-410f-99dc-ca9afd271de1' with status OK
2026.05.06 08:48:36.052571 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.052569 [ 32370 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.052627 [ 32371 ] {} <Debug> DatabaseAtomic (test_13): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.052685 [ 32371 ] {} <Test> DatabaseAtomic (test_13): Counting attached table 02162_test_dictionary to database test_13
2026.05.06 08:48:36.052826 [ 32423 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:36.052878 [ 32371 ] {} <Debug> AsyncLoader: Finish load job 'load table test_13.02162_test_dictionary' with status OK
2026.05.06 08:48:36.052976 [ 32371 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.053193 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT block_excessive_requests_before_settings_ack ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.053365 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT canary_client_privacy                        OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.053472 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT chttp2_batch_requests                        ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.053592 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT chttp2_offload_on_rst_stream                 ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.053638 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT client_privacy                               OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.053683 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT combiner_offload_to_event_engine             ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.053713 [ 32273 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Trace> MergedBlockOutputStream: filled checksums 202605_1_185_37 (state Temporary)
2026.05.06 08:48:36.053726 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT event_engine_client                          OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.053791 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT event_engine_dns                             OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.053839 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT event_engine_listener                        OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.053885 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT free_large_allocator                         OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.053931 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT jitter_max_idle                              ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054034 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT keepalive_fix                                OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054114 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT keepalive_server_fix                         OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054159 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT lazier_stream_updates                        ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054223 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT memory_pressure_controller                   OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054269 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT monitoring_experiment                        ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054316 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT multiping                                    OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054362 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT overload_protection                          ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054418 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT peer_state_based_framing                     OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054464 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT ping_on_rst_stream                           ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054547 [ 32425 ] {} <Test> test_1.test2 (dfcccf80-ca63-4a40-a31e-fafd5aa387b5): loadDataPart: inserting all_1_1_0 (state Active) into data_parts_indexes
2026.05.06 08:48:36.054567 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT promise_based_client_call                    OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054604 [ 32425 ] {} <Trace> test_1.test2 (dfcccf80-ca63-4a40-a31e-fafd5aa387b5): Finished loading Active part all_1_1_0 on disk default
2026.05.06 08:48:36.054643 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT promise_based_server_call                    OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054692 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT red_max_concurrent_streams                   OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054737 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT registered_method_lookup_in_transport        ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054734 [ 32368 ] {} <Debug> test_1.test2 (dfcccf80-ca63-4a40-a31e-fafd5aa387b5): Loaded data parts (1 items)
2026.05.06 08:48:36.054786 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT round_robin_delegate_to_pick_first           ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054834 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT rstpit                                       OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054881 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT schedule_cancellation_over_write             OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.054932 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT separate_ping_from_keepalive                 ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.055010 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT server_privacy                               OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.055083 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT settings_timeout                             ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.055128 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT tarpit                                       ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.055173 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT tcp_frame_size_tuning                        OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.055189 [ 32368 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.055218 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT tcp_rcv_lowat                                OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.055231 [ 32368 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.055266 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT trace_record_callops                         OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.055279 [ 32368 ] {} <Test> DatabaseAtomic (test_1): Counting attached table test2 to database test_1
2026.05.06 08:48:36.055314 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT unconstrained_max_quota_buffer_size          OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.055360 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT work_serializer_dispatch                     OFF (default:OFF) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.055383 [ 32368 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.test2' with status OK
2026.05.06 08:48:36.055438 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT write_size_cap                               ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.055478 [ 32368 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.055497 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT write_size_policy                            ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.055546 [ 31633 ] {} <Debug> grpc: gRPC EXPERIMENT wrr_delegate_to_pick_first                   ON  (default:ON) (contrib/grpc/src/core/lib/experiments/config.cc:183)
2026.05.06 08:48:36.056443 [ 31633 ] {} <Information> grpc: grpc epoll fd: 123 (contrib/grpc/src/core/lib/iomgr/ev_epoll1_linux.cc:123)
2026.05.06 08:48:36.056840 [ 31633 ] {} <Debug> grpc: Using polling engine: epoll1 (contrib/grpc/src/core/lib/iomgr/ev_posix.cc:113)
2026.05.06 08:48:36.057675 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "priority_experimental" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.057865 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "outlier_detection_experimental" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.058019 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "weighted_target_experimental" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.058218 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "pick_first" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.058412 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "round_robin" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.058580 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "weighted_round_robin" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.058890 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "grpclb" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.059342 [ 31633 ] {} <Debug> grpc: Using ares dns resolver (contrib/grpc/src/core/ext/filters/client_channel/resolver/dns/dns_resolver_plugin.cc:50)
2026.05.06 08:48:36.059594 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "rls_experimental" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.060351 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "xds_cluster_manager_experimental" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.060946 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "xds_cluster_impl_experimental" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.061191 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "cds_experimental" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.061916 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "xds_cluster_resolver_experimental" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.062184 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "xds_override_host_experimental" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.062663 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "xds_wrr_locality_experimental" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.063040 [ 31633 ] {} <Debug> grpc: registering LB policy factory for "ring_hash_experimental" (contrib/grpc/src/core/lib/load_balancing/lb_policy_registry.cc:47)
2026.05.06 08:48:36.063265 [ 31633 ] {} <Debug> grpc: registering certificate provider factory for "file_watcher" (contrib/grpc/src/core/lib/security/certificate_provider/certificate_provider_registry.cc:33)
2026.05.06 08:48:36.065809 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading data parts
2026.05.06 08:48:36.066394 [ 32390 ] {} <Test> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.066462 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): There are no data parts
2026.05.06 08:48:36.067020 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_1.txt entry, commands size: 1
2026.05.06 08:48:36.067218 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_5.txt entry, commands size: 1
2026.05.06 08:48:36.067404 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_12.txt entry, commands size: 1
2026.05.06 08:48:36.067591 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_15.txt entry, commands size: 1
2026.05.06 08:48:36.067998 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_14.txt entry, commands size: 1
2026.05.06 08:48:36.068160 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_21.txt entry, commands size: 1
2026.05.06 08:48:36.068304 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_10.txt entry, commands size: 1
2026.05.06 08:48:36.068448 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_16.txt entry, commands size: 1
2026.05.06 08:48:36.068625 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_30.txt entry, commands size: 1
2026.05.06 08:48:36.068775 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_29.txt entry, commands size: 1
2026.05.06 08:48:36.069087 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_35.txt entry, commands size: 1
2026.05.06 08:48:36.069292 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_36.txt entry, commands size: 1
2026.05.06 08:48:36.069431 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_11.txt entry, commands size: 1
2026.05.06 08:48:36.069596 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_22.txt entry, commands size: 1
2026.05.06 08:48:36.069936 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_23.txt entry, commands size: 1
2026.05.06 08:48:36.070237 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_34.txt entry, commands size: 1
2026.05.06 08:48:36.070434 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_17.txt entry, commands size: 1
2026.05.06 08:48:36.070641 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_27.txt entry, commands size: 1
2026.05.06 08:48:36.070800 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_2.txt entry, commands size: 1
2026.05.06 08:48:36.071270 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_18.txt entry, commands size: 1
2026.05.06 08:48:36.071446 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_6.txt entry, commands size: 1
2026.05.06 08:48:36.071495 [ 31633 ] {} <Information> grpc: grpc epoll fd: 56 (contrib/grpc/src/core/lib/event_engine/posix_engine/ev_epoll1_linux.cc:360)
2026.05.06 08:48:36.071671 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_26.txt entry, commands size: 1
2026.05.06 08:48:36.072013 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_31.txt entry, commands size: 1
2026.05.06 08:48:36.072237 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_13.txt entry, commands size: 1
2026.05.06 08:48:36.072572 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_9.txt entry, commands size: 1
2026.05.06 08:48:36.072686 [ 32273 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Renaming temporary part tmp_merge_202605_1_185_37 to 202605_1_185_37 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:36.072761 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_20.txt entry, commands size: 1
2026.05.06 08:48:36.072939 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_19.txt entry, commands size: 1
2026.05.06 08:48:36.072980 [ 32273 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): preparePartForCommit: inserting 202605_1_185_37 (state PreActive) into data_parts_indexes
2026.05.06 08:48:36.073085 [ 32273 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf) (MergerMutator): Merged 5 parts: [202605_1_181_36, 202605_185_185_0] -> 202605_1_185_37
2026.05.06 08:48:36.073104 [ 31633 ] {} <Information> grpc: TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter (contrib/grpc/src/core/lib/iomgr/socket_utils_common_posix.cc:366)
2026.05.06 08:48:36.073120 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_3.txt entry, commands size: 1
2026.05.06 08:48:36.073310 [ 32273 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_1_181_36
2026.05.06 08:48:36.073360 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_32.txt entry, commands size: 1
2026.05.06 08:48:36.073440 [ 32273 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_182_182_0
2026.05.06 08:48:36.073518 [ 32273 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_183_183_0
2026.05.06 08:48:36.073582 [ 32273 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_184_184_0
2026.05.06 08:48:36.073604 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_8.txt entry, commands size: 1
2026.05.06 08:48:36.073644 [ 32273 ] {ba5f8386-a6a6-42b6-bc2d-5b7408db06cf::202605_1_185_37} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf), part: 202605_185_185_0
2026.05.06 08:48:36.073802 [ 31633 ] {} <Information> Application: Listening for gRPC protocol: [::]:9100
2026.05.06 08:48:36.073820 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_24.txt entry, commands size: 1
2026.05.06 08:48:36.073934 [ 31633 ] {} <Information> Application: Listening for Prometheus: http://[::]:9988
2026.05.06 08:48:36.074014 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_7.txt entry, commands size: 1
2026.05.06 08:48:36.074043 [ 31633 ] {} <Information> ZooKeeperClient: ZK send fault: 0.002%
2026.05.06 08:48:36.074512 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_25.txt entry, commands size: 1
2026.05.06 08:48:36.074556 [ 31633 ] {} <Information> ZooKeeperClient: ZK recv fault: 0.002%
2026.05.06 08:48:36.074715 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_33.txt entry, commands size: 1
2026.05.06 08:48:36.074705 [ 31633 ] {} <Information> ZooKeeperClient: ZK send sleep: 0.001% -> 10000ms
2026.05.06 08:48:36.074840 [ 31633 ] {} <Information> Application: Ready for connections.
2026.05.06 08:48:36.074869 [ 32273 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 19.72 MiB.
2026.05.06 08:48:36.074914 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_4.txt entry, commands size: 1
2026.05.06 08:48:36.075108 [ 32390 ] {} <Debug> test_1.concurrent_alter_column (8aae60fd-3375-446a-8066-397df7275956): Loading mutation: mutation_28.txt entry, commands size: 1
2026.05.06 08:48:36.076190 [ 32390 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.076250 [ 32390 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.076316 [ 32390 ] {} <Test> DatabaseAtomic (test_1): Counting attached table concurrent_alter_column to database test_1
2026.05.06 08:48:36.076956 [ 32390 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.concurrent_alter_column' with status OK
2026.05.06 08:48:36.077027 [ 32390 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.079026 [ 32339 ] {} <Debug> test.hits_dst (28471a99-32bc-48d3-8d12-320f641b9d91): Loading data parts
2026.05.06 08:48:36.079435 [ 32339 ] {} <Test> test.hits_dst (28471a99-32bc-48d3-8d12-320f641b9d91): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.079492 [ 32339 ] {} <Debug> test.hits_dst (28471a99-32bc-48d3-8d12-320f641b9d91): There are no data parts
2026.05.06 08:48:36.080738 [ 32339 ] {} <Debug> DatabaseAtomic (test): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.080799 [ 32339 ] {} <Debug> DatabaseAtomic (test): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.080844 [ 32339 ] {} <Test> DatabaseAtomic (test): Counting attached table hits_dst to database test
2026.05.06 08:48:36.081760 [ 32339 ] {} <Debug> AsyncLoader: Finish load job 'load table test.hits_dst' with status OK
2026.05.06 08:48:36.081807 [ 32339 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.082526 [ 32401 ] {} <Debug> test.hits_s3 (6c319968-5265-45e4-8dab-857e51b8d284): Loading data parts
2026.05.06 08:48:36.082877 [ 32402 ] {} <Debug> test.hits (81954dcf-0215-45ea-9a1d-25d73877ca18): Loading data parts
2026.05.06 08:48:36.082907 [ 32401 ] {} <Test> test.hits_s3 (6c319968-5265-45e4-8dab-857e51b8d284): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.082970 [ 32401 ] {} <Debug> test.hits_s3 (6c319968-5265-45e4-8dab-857e51b8d284): There are no data parts
2026.05.06 08:48:36.084349 [ 32401 ] {} <Debug> DatabaseAtomic (test): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.084418 [ 32401 ] {} <Debug> DatabaseAtomic (test): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.084492 [ 32401 ] {} <Test> DatabaseAtomic (test): Counting attached table hits_s3 to database test
2026.05.06 08:48:36.085326 [ 32397 ] {} <Debug> test.visits (46d6b65c-6d48-44b5-8244-0a632c789d94): Loading data parts
2026.05.06 08:48:36.085464 [ 32401 ] {} <Debug> AsyncLoader: Finish load job 'load table test.hits_s3' with status OK
2026.05.06 08:48:36.085546 [ 32401 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.090387 [ 32402 ] {} <Test> test.hits (81954dcf-0215-45ea-9a1d-25d73877ca18): loadDataParts: clearing data_parts_indexes (had 0 parts)
2026.05.06 08:48:36.090458 [ 32402 ] {} <Debug> test.hits (81954dcf-0215-45ea-9a1d-25d73877ca18): There are no data parts
2026.05.06 08:48:36.091742 [ 32402 ] {} <Debug> DatabaseAtomic (test): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.091788 [ 32402 ] {} <Debug> DatabaseAtomic (test): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.091840 [ 32402 ] {} <Test> DatabaseAtomic (test): Counting attached table hits to database test
2026.05.06 08:48:36.092791 [ 32402 ] {} <Debug> AsyncLoader: Finish load job 'load table test.hits' with status OK
2026.05.06 08:48:36.092850 [ 32402 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.125574 [ 32423 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: 03008_test_s3_mt/dklbspircelrvhncgiwwqcgkngqcfwkp/default_compression_codec.txt (store/fef/fef9e575-f89d-43f4-933c-47fb46bf5de1/0_1_1_1/default_compression_codec.txt)
2026.05.06 08:48:36.131347 [ 32423 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:36.131651 [ 32423 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:36 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE728FCD1AEA0
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:36.131901 [ 32423 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:36.134424 [ 32423 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:36.134648 [ 32423 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:36 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE728FD01174E
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:36.134874 [ 32423 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:36.135368 [ 32423 ] {} <Test> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): loadDataPart: inserting 0_1_1_1 (state Active) into data_parts_indexes
2026.05.06 08:48:36.135436 [ 32423 ] {} <Trace> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): Finished loading Active part 0_1_1_1 on disk 03008_s3_plain_rewritable
2026.05.06 08:48:36.135729 [ 32353 ] {} <Debug> test_1.test_s3_mt (fef9e575-f89d-43f4-933c-47fb46bf5de1): Loaded data parts (1 items)
2026.05.06 08:48:36.139958 [ 32353 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:36.140416 [ 32353 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:36 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE728FD549BE7
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:36.140771 [ 32353 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:36.144725 [ 32353 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:36.145193 [ 32353 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: ::1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:36 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE728FD9C0B40
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:36.145565 [ 32353 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:36.146932 [ 32353 ] {} <Debug> DatabaseAtomic (test_1): There are 0 detached tables. Start searching non used tables.
2026.05.06 08:48:36.147035 [ 32353 ] {} <Debug> DatabaseAtomic (test_1): Found 0 non used tables in detached tables.
2026.05.06 08:48:36.147157 [ 32353 ] {} <Test> DatabaseAtomic (test_1): Counting attached table test_s3_mt to database test_1
2026.05.06 08:48:36.147469 [ 32353 ] {} <Debug> AsyncLoader: Finish load job 'load table test_1.test_s3_mt' with status OK
2026.05.06 08:48:36.147649 [ 32353 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.154738 [ 32481 ] {} <Test> Poco: Queue size: 0, current threads: 0, threads in pool: 3, current connections: 0
2026.05.06 08:48:36.155092 [ 31641 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:38146
2026.05.06 08:48:36.159676 [ 31641 ] {} <Debug> TCPHandler: Connected ClickHouse client version 24.8.0, revision: 54472, user: default.
2026.05.06 08:48:36.160094 [ 31641 ] {} <Debug> TCP-Session-8a8946d9-f5d4-4a31-98c4-742641c11169: Authenticating user 'default' from [::1]:38146
2026.05.06 08:48:36.160807 [ 31641 ] {} <Debug> TCP-Session-8a8946d9-f5d4-4a31-98c4-742641c11169: 8a8946d9-f5d4-4a31-98c4-742641c11169 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:36.161112 [ 31641 ] {} <Debug> TCP-Session-8a8946d9-f5d4-4a31-98c4-742641c11169: Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:36.165845 [ 31641 ] {} <Trace> ContextAccess (default): Settings: readonly = 0, allow_ddl = true, allow_introspection_functions = true
2026.05.06 08:48:36.166096 [ 31641 ] {} <Trace> ContextAccess (default): List of all grants: GRANT ALL ON *.* WITH GRANT OPTION
2026.05.06 08:48:36.166285 [ 31641 ] {} <Trace> ContextAccess (default): List of all grants including implicit: GRANT ALL ON *.* WITH GRANT OPTION
2026.05.06 08:48:36.205442 [ 31641 ] {} <Trace> TCP-Session-8a8946d9-f5d4-4a31-98c4-742641c11169: 8a8946d9-f5d4-4a31-98c4-742641c11169 Creating query context from session context, user_id: 94309d50-4f52-5250-31bd-74fecac179db, parent context user: default
2026.05.06 08:48:36.207853 [ 31641 ] {23423ee7-e90d-469e-8ea9-15e322ed5d79} <Debug> executeQuery: (from [::1]:38146) SELECT 1 (stage: Complete)
2026.05.06 08:48:36.211470 [ 31641 ] {23423ee7-e90d-469e-8ea9-15e322ed5d79} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:36.213103 [ 31641 ] {23423ee7-e90d-469e-8ea9-15e322ed5d79} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:36.221597 [ 31641 ] {23423ee7-e90d-469e-8ea9-15e322ed5d79} <Debug> executeQuery: Read 1 rows, 1.00 B in 0.01418 sec., 70.52186177715092 rows/sec., 70.52 B/sec.
2026.05.06 08:48:36.222800 [ 31641 ] {23423ee7-e90d-469e-8ea9-15e322ed5d79} <Debug> TCPHandler: Processed in 0.018438413 sec.
2026.05.06 08:48:36.310404 [ 31641 ] {} <Test> TCPHandler: Closing connection (open: true, cancelled: false, eof: true)
2026.05.06 08:48:36.310586 [ 31641 ] {} <Debug> TCPHandler: Done processing connection.
2026.05.06 08:48:36.311791 [ 31641 ] {} <Debug> TCP-Session-8a8946d9-f5d4-4a31-98c4-742641c11169: 8a8946d9-f5d4-4a31-98c4-742641c11169 Logout, user_id: 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:36.476405 [ 32481 ] {} <Test> Poco: Queue size: 0, current threads: 1, threads in pool: 3, current connections: 0
2026.05.06 08:48:36.476754 [ 31641 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:38154
2026.05.06 08:48:36.481172 [ 31641 ] {} <Debug> TCPHandler: Connected ClickHouse client version 24.8.0, revision: 54472, user: default.
2026.05.06 08:48:36.481365 [ 31641 ] {} <Debug> TCP-Session-50c31820-497d-459e-a5ab-c5bf6fa935e8: Authenticating user 'default' from [::1]:38154
2026.05.06 08:48:36.481694 [ 31641 ] {} <Debug> TCP-Session-50c31820-497d-459e-a5ab-c5bf6fa935e8: 50c31820-497d-459e-a5ab-c5bf6fa935e8 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:36.481806 [ 31641 ] {} <Debug> TCP-Session-50c31820-497d-459e-a5ab-c5bf6fa935e8: Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:36.520492 [ 32397 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=1, jobs=1): Cannot attach table `test`.`visits` from metadata file /var/lib/clickhouse/store/3ce/3cedaf24-7715-4cb7-ac56-c3bafb2a379b/visits.sql from query ATTACH TABLE test.visits UUID '46d6b65c-6d48-44b5-8244-0a632c789d94' (`CounterID` UInt32, `StartDate` Date, `Sign` Int8, `IsNew` UInt8, `VisitID` UInt64, `UserID` UInt64, `StartTime` DateTime, `Duration` UInt32, `UTCStartTime` DateTime, `PageViews` Int32, `Hits` Int32, `IsBounce` UInt8, `Referer` String, `StartURL` String, `RefererDomain` String, `StartURLDomain` String, `EndURL` String, `LinkURL` String, `IsDownload` UInt8, `TraficSourceID` Int8, `SearchEngineID` UInt16, `SearchPhrase` String, `AdvEngineID` UInt8, `PlaceID` Int32, `RefererCategories` Array(UInt16), `URLCategories` Array(UInt16), `URLRegions` Array(UInt32), `RefererRegions` Array(UInt32), `IsYandex` UInt8, `GoalReachesDepth` Int32, `GoalReachesURL` Int32, `GoalReachesAny` Int32, `SocialSourceNetworkID` UInt8, `SocialSourcePage` String, `MobilePhoneModel` String, `ClientEventTime` DateTime, `RegionID` UInt32, `ClientIP` UInt32, `ClientIP6` FixedString(16), `RemoteIP` UInt32, `RemoteIP6` FixedString(16), `IPNetworkID` UInt32, `SilverlightVersion3` UInt32, `CodeVersion` UInt32, `ResolutionWidth` UInt16, `ResolutionHeight` UInt16, `UserAgentMajor` UInt16, `UserAgentMinor` UInt16, `WindowClientWidth` UInt16, `WindowClientHeight` UInt16, `SilverlightVersion2` UInt8, `SilverlightVersion4` UInt16, `FlashVersion3` UInt16, `FlashVersion4` UInt16, `ClientTimeZone` Int16, `OS` UInt8, `UserAgent` UInt8, `ResolutionDepth` UInt8, `FlashMajor` UInt8, `FlashMinor` UInt8, `NetMajor` UInt8, `NetMinor` UInt8, `MobilePhone` UInt8, `SilverlightVersion1` UInt8, `Age` UInt8, `Sex` UInt8, `Income` UInt8, `JavaEnable` UInt8, `CookieEnable` UInt8, `JavascriptEnable` UInt8, `IsMobile` UInt8, `BrowserLanguage` UInt16, `BrowserCountry` UInt16, `Interests` UInt16, `Robotness` UInt8, `GeneralInterests` Array(UInt16), `Params` Array(String), `Goals.ID` Array(UInt32), `Goals.Serial` Array(UInt32), `Goals.EventTime` Array(DateTime), `Goals.Price` Array(Int64), `Goals.OrderID` Array(String), `Goals.CurrencyID` Array(UInt32), `WatchIDs` Array(UInt64), `ParamSumPrice` Int64, `ParamCurrency` FixedString(3), `ParamCurrencyID` UInt16, `ClickLogID` UInt64, `ClickEventID` Int32, `ClickGoodEvent` Int32, `ClickEventTime` DateTime, `ClickPriorityID` Int32, `ClickPhraseID` Int32, `ClickPageID` Int32, `ClickPlaceID` Int32, `ClickTypeID` Int32, `ClickResourceID` Int32, `ClickCost` UInt32, `ClickClientIP` UInt32, `ClickDomainID` UInt32, `ClickURL` String, `ClickAttempt` UInt8, `ClickOrderID` UInt32, `ClickBannerID` UInt32, `ClickMarketCategoryID` UInt32, `ClickMarketPP` UInt32, `ClickMarketCategoryName` String, `ClickMarketPPName` String, `ClickAWAPSCampaignName` String, `ClickPageName` String, `ClickTargetType` UInt16, `ClickTargetPhraseID` UInt64, `ClickContextType` UInt8, `ClickSelectType` Int8, `ClickOptions` String, `ClickGroupBannerID` Int32, `OpenstatServiceName` String, `OpenstatCampaignID` String, `OpenstatAdID` String, `OpenstatSourceID` String, `UTMSource` String, `UTMMedium` String, `UTMCampaign` String, `UTMContent` String, `UTMTerm` String, `FromTag` String, `HasGCLID` UInt8, `FirstVisit` DateTime, `PredLastVisit` Date, `LastVisit` Date, `TotalVisits` UInt32, `TraficSource.ID` Array(Int8), `TraficSource.SearchEngineID` Array(UInt16), `TraficSource.AdvEngineID` Array(UInt8), `TraficSource.PlaceID` Array(UInt16), `TraficSource.SocialSourceNetworkID` Array(UInt8), `TraficSource.Domain` Array(String), `TraficSource.SearchPhrase` Array(String), `TraficSource.SocialSourcePage` Array(String), `Attendance` FixedString(16), `CLID` UInt32, `YCLID` UInt64, `NormalizedRefererHash` UInt64, `SearchPhraseHash` UInt64, `RefererDomainHash` UInt64, `NormalizedStartURLHash` UInt64, `StartURLDomainHash` UInt64, `NormalizedEndURLHash` UInt64, `TopLevelDomain` UInt64, `URLScheme` UInt64, `OpenstatServiceNameHash` UInt64, `OpenstatCampaignIDHash` UInt64, `OpenstatAdIDHash` UInt64, `OpenstatSourceIDHash` UInt64, `UTMSourceHash` UInt64, `UTMMediumHash` UInt64, `UTMCampaignHash` UInt64, `UTMContentHash` UInt64, `UTMTermHash` UInt64, `FromHash` UInt64, `WebVisorEnabled` UInt8, `WebVisorActivity` UInt32, `ParsedParams.Key1` Array(String), `ParsedParams.Key2` Array(String), `ParsedParams.Key3` Array(String), `ParsedParams.Key4` Array(String), `ParsedParams.Key5` Array(String), `ParsedParams.ValueDouble` Array(Float64), `Market.Type` Array(UInt8), `Market.GoalID` Array(UInt32), `Market.OrderID` Array(String), `Market.OrderPrice` Array(Int64), `Market.PP` Array(UInt32), `Market.DirectPlaceID` Array(UInt32), `Market.DirectOrderID` Array(UInt32), `Market.DirectBannerID` Array(UInt32), `Market.GoodID` Array(String), `Market.GoodName` Array(String), `Market.GoodQuantity` Array(Int32), `Market.GoodPrice` Array(Int64), `IslandID` FixedString(16)) ENGINE = CollapsingMergeTree(Sign) PARTITION BY toYYYYMM(StartDate) ORDER BY (CounterID, StartDate, intHash32(UserID), VisitID) SAMPLE BY intHash32(UserID) SETTINGS index_granularity = 8192, storage_policy = 'default'. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./src/Common/threadPoolCallbackRunner.h:0: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000030516e9d
7. ./build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:0: DB::MergeTreeData::loadDataParts(bool, std::optional<std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>>) @ 0x00000000398457e5
8. ./contrib/llvm-project/libcxx/include/optional:260: DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, DB::LoadingStrictnessLevel, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>) @ 0x000000003a4211ea
9. ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:0: std::shared_ptr<DB::StorageMergeTree> std::allocate_shared[abi:v15007]<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(std::allocator<DB::StorageMergeTree> const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000003a41f2ef
10. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:0: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000003a40e303
11. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x0000000030da3bba
12. ./build_docker/./src/Storages/StorageFactory.cpp:225: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x00000000384436a6
13. ./build_docker/./src/Databases/DatabaseOnDisk.cpp:0: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x0000000032a776dd
14. ./contrib/llvm-project/libcxx/include/string:1499: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x0000000032af6431
15. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x0000000032b0fe9b
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x0000000020718306
17. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x0000000020719d20
18. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
19. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
20. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
21. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
22. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
23. ? @ 0x00007f6e2fbe4ac3
24. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:36.522807 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'load table test.visits' with status FAILED
2026.05.06 08:48:36.523433 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'startup ddl worker' with status CANCELED
2026.05.06 08:48:36.524260 [ 32397 ] {} <Debug> AsyncLoader: Finish load job 'startup table test.visits' with status CANCELED
2026.05.06 08:48:36.524286 [ 31641 ] {} <Trace> TCP-Session-50c31820-497d-459e-a5ab-c5bf6fa935e8: 50c31820-497d-459e-a5ab-c5bf6fa935e8 Creating query context from session context, user_id: 94309d50-4f52-5250-31bd-74fecac179db, parent context user: default
2026.05.06 08:48:36.524327 [ 32397 ] {} <Debug> AsyncLoader: Stop worker in BackgroundLoad
2026.05.06 08:48:36.524420 [ 32397 ] {} <Debug> AsyncLoader: Change current priority: 1 -> 2
2026.05.06 08:48:36.524498 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #1 in BackgrndStartup
2026.05.06 08:48:36.524618 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #2 in BackgrndStartup
2026.05.06 08:48:36.524671 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #3 in BackgrndStartup
2026.05.06 08:48:36.524854 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #4 in BackgrndStartup
2026.05.06 08:48:36.524979 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #5 in BackgrndStartup
2026.05.06 08:48:36.525022 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #6 in BackgrndStartup
2026.05.06 08:48:36.525106 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #7 in BackgrndStartup
2026.05.06 08:48:36.525151 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #8 in BackgrndStartup
2026.05.06 08:48:36.525274 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #9 in BackgrndStartup
2026.05.06 08:48:36.525322 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #10 in BackgrndStartup
2026.05.06 08:48:36.525406 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #11 in BackgrndStartup
2026.05.06 08:48:36.525448 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #12 in BackgrndStartup
2026.05.06 08:48:36.525604 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #13 in BackgrndStartup
2026.05.06 08:48:36.525646 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #14 in BackgrndStartup
2026.05.06 08:48:36.525696 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #15 in BackgrndStartup
2026.05.06 08:48:36.525761 [ 32397 ] {} <Debug> AsyncLoader: Spawn loader worker #16 in BackgrndStartup
2026.05.06 08:48:36.525925 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database default' in BackgrndStartup
2026.05.06 08:48:36.526105 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database default' with status OK
2026.05.06 08:48:36.526256 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database rdb_test_13_3' in BackgrndStartup
2026.05.06 08:48:36.526373 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database rdb_test_13_3' with status OK
2026.05.06 08:48:36.526511 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database rdb_test_13_9' in BackgrndStartup
2026.05.06 08:48:36.526621 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database rdb_test_13_9' with status OK
2026.05.06 08:48:36.526727 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database datasets' in BackgrndStartup
2026.05.06 08:48:36.526833 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database datasets' with status OK
2026.05.06 08:48:36.526934 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_11' in BackgrndStartup
2026.05.06 08:48:36.527034 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_11' with status OK
2026.05.06 08:48:36.527191 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database rdb_test_13_2' in BackgrndStartup
2026.05.06 08:48:36.527336 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database rdb_test_13_2' with status OK
2026.05.06 08:48:36.527442 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.t01565' in BackgrndStartup
2026.05.06 08:48:36.527667 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_5' in BackgrndStartup
2026.05.06 08:48:36.527925 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_5' with status OK
2026.05.06 08:48:36.528080 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.input_function_table_2' in BackgrndStartup
2026.05.06 08:48:36.528205 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.data2014' in BackgrndStartup
2026.05.06 08:48:36.528304 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.data2014' with status OK
2026.05.06 08:48:36.528370 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.s' in BackgrndStartup
2026.05.06 08:48:36.528469 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.s' with status OK
2026.05.06 08:48:36.528553 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.tmp_01781' in BackgrndStartup
2026.05.06 08:48:36.528629 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.tmp_01781' with status OK
2026.05.06 08:48:36.528685 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.mv' in BackgrndStartup
2026.05.06 08:48:36.528745 [ 31641 ] {696a4bd3-28ea-4388-b32e-930c9288432f} <Debug> executeQuery: (from [::1]:38154) SELECT count() FROM system.build_options WHERE name = 'CXX_FLAGS' AND position('sanitize=address' IN value) (stage: Complete)
2026.05.06 08:48:36.528831 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.decimal' in BackgrndStartup
2026.05.06 08:48:36.529057 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.decimal' with status OK
2026.05.06 08:48:36.529193 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.datetime_t' in BackgrndStartup
2026.05.06 08:48:36.529370 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.source_table_tiny_log' in BackgrndStartup
2026.05.06 08:48:36.529604 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.source_table_tiny_log' with status OK
2026.05.06 08:48:36.529764 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_xfltqef0.buf' in BackgrndStartup
2026.05.06 08:48:36.529960 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.dest_view_stripe_log' in BackgrndStartup
2026.05.06 08:48:36.530139 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.dest_view_stripe_log' with status OK
2026.05.06 08:48:36.530266 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.dst' in BackgrndStartup
2026.05.06 08:48:36.530391 [ 32439 ] {} <Trace> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Starting up table
2026.05.06 08:48:36.530417 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup table shard_1.from_0' in BackgrndStartup
2026.05.06 08:48:36.530587 [ 32424 ] {} <Trace> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Starting up table
2026.05.06 08:48:36.530641 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.data' in BackgrndStartup
2026.05.06 08:48:36.530803 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.data' with status OK
2026.05.06 08:48:36.530902 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.test_table' in BackgrndStartup
2026.05.06 08:48:36.531040 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.test_table' with status OK
2026.05.06 08:48:36.531162 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.dest_view_memory' in BackgrndStartup
2026.05.06 08:48:36.531262 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.dest_view_memory' with status OK
2026.05.06 08:48:36.531344 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.collapsing_suspicious_granularity' in BackgrndStartup
2026.05.06 08:48:36.531476 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.input_function_table_2' with status OK
2026.05.06 08:48:36.531689 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.t_r_ok' in BackgrndStartup
2026.05.06 08:48:36.531823 [ 32438 ] {} <Trace> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Starting up table
2026.05.06 08:48:36.531875 [ 32459 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.t' in BackgrndStartup
2026.05.06 08:48:36.531996 [ 32459 ] {} <Trace> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Starting up table
2026.05.06 08:48:36.532046 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.source_table_memory' in BackgrndStartup
2026.05.06 08:48:36.532286 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.source_table_memory' with status OK
2026.05.06 08:48:36.532391 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.agg' in BackgrndStartup
2026.05.06 08:48:36.532517 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.mv' with status OK
2026.05.06 08:48:36.532594 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.test_tuple_filter__fuzz_2' in BackgrndStartup
2026.05.06 08:48:36.532715 [ 32442 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_mkf6rch8.test_02481_mismatch_files' in BackgrndStartup
2026.05.06 08:48:36.532860 [ 32442 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_mkf6rch8.test_02481_mismatch_files' with status OK
2026.05.06 08:48:36.532969 [ 32442 ] {} <Debug> AsyncLoader: Execute load job 'startup table shard_0.from_0' in BackgrndStartup
2026.05.06 08:48:36.533051 [ 32442 ] {} <Trace> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Starting up table
2026.05.06 08:48:36.533120 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.datetime_t' with status OK
2026.05.06 08:48:36.533274 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_01516.primary_key_test' in BackgrndStartup
2026.05.06 08:48:36.533522 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_xfltqef0.buf' with status OK
2026.05.06 08:48:36.533632 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test..inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8' in BackgrndStartup
2026.05.06 08:48:36.533821 [ 32342 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1..inner_id.8ca14fb2-e48e-4ec8-8fdf-854e1dc4cf8c' in BackgrndStartup
2026.05.06 08:48:36.534070 [ 32462 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.partslost_1' in BackgrndStartup
2026.05.06 08:48:36.534197 [ 32462 ] {} <Trace> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Starting up table
2026.05.06 08:48:36.534231 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.t01565' with status OK
2026.05.06 08:48:36.534257 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 110
2026.05.06 08:48:36.534369 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.table_gcd_codec_one_hundred_zeros' in BackgrndStartup
2026.05.06 08:48:36.534549 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.534566 [ 32467 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.test_table_comment' in BackgrndStartup
2026.05.06 08:48:36.534740 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.collapsing_suspicious_granularity' with status OK
2026.05.06 08:48:36.534934 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.test_without_merge' in BackgrndStartup
2026.05.06 08:48:36.535044 [ 32477 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Starting up table
2026.05.06 08:48:36.535081 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 83
2026.05.06 08:48:36.535085 [ 32344 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.input_function_table_3' in BackgrndStartup
2026.05.06 08:48:36.535219 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.535251 [ 32436 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.src' in BackgrndStartup
2026.05.06 08:48:36.535364 [ 32436 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Starting up table
2026.05.06 08:48:36.535458 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.agg' with status OK
2026.05.06 08:48:36.535638 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.mv_b_even_even' in BackgrndStartup
2026.05.06 08:48:36.535758 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 62
2026.05.06 08:48:36.535758 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.test_tuple_filter__fuzz_2' with status OK
2026.05.06 08:48:36.535914 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.dest_view_merge_tree' in BackgrndStartup
2026.05.06 08:48:36.535931 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.535993 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.dest_view_merge_tree' with status OK
2026.05.06 08:48:36.536047 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.numbers_squashed' in BackgrndStartup
2026.05.06 08:48:36.536107 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.numbers_squashed' with status OK
2026.05.06 08:48:36.536154 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.A' in BackgrndStartup
2026.05.06 08:48:36.536211 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.A' with status OK
2026.05.06 08:48:36.536257 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.constrained' in BackgrndStartup
2026.05.06 08:48:36.536311 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.constrained' with status OK
2026.05.06 08:48:36.536325 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 62
2026.05.06 08:48:36.536359 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.mv' in BackgrndStartup
2026.05.06 08:48:36.536465 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.536522 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.table_gcd_codec_one_hundred_zeros' with status OK
2026.05.06 08:48:36.536709 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.B' in BackgrndStartup
2026.05.06 08:48:36.536860 [ 32344 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.input_function_table_3' with status OK
2026.05.06 08:48:36.537009 [ 32344 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.t2' in BackgrndStartup
2026.05.06 08:48:36.537189 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.mv_b_even_even' with status OK
2026.05.06 08:48:36.537298 [ 31641 ] {696a4bd3-28ea-4388-b32e-930c9288432f} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:36.537318 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_w50hoezt.test_constraints' in BackgrndStartup
2026.05.06 08:48:36.537468 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_01516.primary_key_test' with status OK
2026.05.06 08:48:36.537615 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.merge_tree' in BackgrndStartup
2026.05.06 08:48:36.537633 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 83
2026.05.06 08:48:36.537841 [ 32342 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1..inner_id.8ca14fb2-e48e-4ec8-8fdf-854e1dc4cf8c' with status OK
2026.05.06 08:48:36.537980 [ 32342 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.table_dst' in BackgrndStartup
2026.05.06 08:48:36.538047 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.538164 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test..inner_id.3d822c25-1d74-4d5a-b82e-8bbcb109d0e8' with status OK
2026.05.06 08:48:36.538117 [ 31870 ] {} <Debug> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95) (MergerMutator): Selected 4 parts from all_1_1_0 to all_4_4_0
2026.05.06 08:48:36.538339 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table shard_0.data_02346' in BackgrndStartup
2026.05.06 08:48:36.538509 [ 32467 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.test_table_comment' with status OK
2026.05.06 08:48:36.538529 [ 31870 ] {} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:36.538651 [ 32467 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.t0' in BackgrndStartup
2026.05.06 08:48:36.538723 [ 31870 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:36.538773 [ 32342 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.table_dst' with status OK
2026.05.06 08:48:36.538994 [ 32342 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.ps' in BackgrndStartup
2026.05.06 08:48:36.539123 [ 32342 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.ps' with status OK
2026.05.06 08:48:36.539133 [ 31641 ] {696a4bd3-28ea-4388-b32e-930c9288432f} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:36.539212 [ 32342 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_2gxskyku_1.t' in BackgrndStartup
2026.05.06 08:48:36.539306 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 90
2026.05.06 08:48:36.539313 [ 32342 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_2gxskyku_1.t' with status OK
2026.05.06 08:48:36.539462 [ 32342 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.array_jaccard_index' in BackgrndStartup
2026.05.06 08:48:36.539625 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.539658 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_w50hoezt.test_constraints' with status OK
2026.05.06 08:48:36.539716 [ 31873 ] {} <Debug> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d) (MergerMutator): Selected 16 parts from all_1_1_0 to all_16_16_0
2026.05.06 08:48:36.539795 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.test_table_2' in BackgrndStartup
2026.05.06 08:48:36.539835 [ 32284 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTask::PrepareStage: Merging 4 parts: from all_1_1_0 to all_4_4_0 into Wide with storage Full
2026.05.06 08:48:36.540048 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.merge_tree' with status OK
2026.05.06 08:48:36.540092 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 2, bytes: 206
2026.05.06 08:48:36.540114 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.test_collation' in BackgrndStartup
2026.05.06 08:48:36.540173 [ 31874 ] {} <Debug> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b) (MergerMutator): Selected 3 parts from all_1_1_0 to all_3_3_0
2026.05.06 08:48:36.540260 [ 32467 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.t0' with status OK
2026.05.06 08:48:36.540388 [ 32467 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.t1' in BackgrndStartup
2026.05.06 08:48:36.540433 [ 31874 ] {} <Trace> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:36.540442 [ 32244 ] {} <Test> KeeperLogStore: Flushing 2 logs
2026.05.06 08:48:36.540428 [ 31873 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:36.540501 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.mv' with status OK
2026.05.06 08:48:36.540559 [ 31874 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:36.540744 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.t_index_hypothesis' in BackgrndStartup
2026.05.06 08:48:36.540881 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table shard_0.data_02346' with status OK
2026.05.06 08:48:36.541002 [ 32284 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Vertical
2026.05.06 08:48:36.541072 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.source_table_merge_tree' in BackgrndStartup
2026.05.06 08:48:36.541043 [ 31878 ] {} <Debug> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Loading 5 outdated data parts asynchronously
2026.05.06 08:48:36.541014 [ 31873 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:36.541239 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.B' with status OK
2026.05.06 08:48:36.541376 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.dim_model' in BackgrndStartup
2026.05.06 08:48:36.541509 [ 32473 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Loading Outdated part all_5_5_0 from disk default
2026.05.06 08:48:36.541605 [ 32465 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Loading Outdated part all_3_3_0 from disk default
2026.05.06 08:48:36.541670 [ 32344 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.t2' with status OK
2026.05.06 08:48:36.541651 [ 32422 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Loading Outdated part all_4_4_0 from disk default
2026.05.06 08:48:36.541708 [ 32434 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Loading Outdated part all_1_1_0 from disk default
2026.05.06 08:48:36.541732 [ 32344 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.defaulted' in BackgrndStartup
2026.05.06 08:48:36.541705 [ 32460 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Loading Outdated part all_2_2_0 from disk default
2026.05.06 08:48:36.541793 [ 32344 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.defaulted' with status OK
2026.05.06 08:48:36.541835 [ 32344 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.table_for_rename_replicated' in BackgrndStartup
2026.05.06 08:48:36.541886 [ 32344 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Starting up table
2026.05.06 08:48:36.541946 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.test_table_2' with status OK
2026.05.06 08:48:36.542363 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.json_test' in BackgrndStartup
2026.05.06 08:48:36.542534 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.test_collation' with status OK
2026.05.06 08:48:36.542598 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.test1' in BackgrndStartup
2026.05.06 08:48:36.542591 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTask::PrepareStage: Merging 16 parts: from all_1_1_0 to all_16_16_0 into Compact with storage Full
2026.05.06 08:48:36.542694 [ 32467 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.t1' with status OK
2026.05.06 08:48:36.542771 [ 32286 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Debug> MergeTask::PrepareStage: Merging 3 parts: from all_1_1_0 to all_3_3_0 into Compact with storage Full
2026.05.06 08:48:36.542821 [ 32467 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.table_for_freeze' in BackgrndStartup
2026.05.06 08:48:36.543194 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.t_index_hypothesis' with status OK
2026.05.06 08:48:36.544413 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.rename_table_multiple' in BackgrndStartup
2026.05.06 08:48:36.545082 [ 32434 ] {} <Test> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): loadDataPart: inserting all_1_1_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.545149 [ 32434 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Finished loading Outdated part all_1_1_0 on disk default
2026.05.06 08:48:36.545517 [ 32434 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b), part: all_1_1_0
2026.05.06 08:48:36.545641 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.dim_model' with status OK
2026.05.06 08:48:36.545799 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_m91yt8r2_1.t' in BackgrndStartup
2026.05.06 08:48:36.545967 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.source_table_merge_tree' with status OK
2026.05.06 08:48:36.546087 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.h3_indexes' in BackgrndStartup
2026.05.06 08:48:36.546221 [ 32342 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.array_jaccard_index' with status OK
2026.05.06 08:48:36.546221 [ 32465 ] {} <Test> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): loadDataPart: inserting all_3_3_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.546343 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:36.546370 [ 32342 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.mv_b_even' in BackgrndStartup
2026.05.06 08:48:36.546413 [ 32465 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Finished loading Outdated part all_3_3_0 on disk default
2026.05.06 08:48:36.546535 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.test1' with status OK
2026.05.06 08:48:36.546729 [ 32422 ] {} <Test> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): loadDataPart: inserting all_4_4_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.546867 [ 32422 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Finished loading Outdated part all_4_4_0 on disk default
2026.05.06 08:48:36.547818 [ 31889 ] {} <Debug> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92) (MergerMutator): Selected 3 parts from all_1_1_0_6 to all_5_5_0_6
2026.05.06 08:48:36.547832 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.548029 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 139
2026.05.06 08:48:36.548756 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.t1__fuzz_26' in BackgrndStartup
2026.05.06 08:48:36.548766 [ 32460 ] {} <Test> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): loadDataPart: inserting all_2_2_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.548828 [ 32465 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b), part: all_3_3_0
2026.05.06 08:48:36.548830 [ 32286 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:36.548873 [ 31889 ] {} <Trace> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:36.548895 [ 32422 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b), part: all_4_4_0
2026.05.06 08:48:36.548947 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.json_test' with status OK
2026.05.06 08:48:36.548972 [ 32460 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Finished loading Outdated part all_2_2_0 on disk default
2026.05.06 08:48:36.549008 [ 31889 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:36.549092 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.test' in BackgrndStartup
2026.05.06 08:48:36.549197 [ 32473 ] {} <Test> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): loadDataPart: inserting all_5_5_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.549218 [ 32286 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.549263 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.549276 [ 32473 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Finished loading Outdated part all_5_5_0 on disk default
2026.05.06 08:48:36.549277 [ 32362 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.549402 [ 32473 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b), part: all_5_5_0
2026.05.06 08:48:36.549441 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.t1__fuzz_26' with status OK
2026.05.06 08:48:36.549536 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.549609 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.t_proj_external' in BackgrndStartup
2026.05.06 08:48:36.549256 [ 32460 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b), part: all_2_2_0
2026.05.06 08:48:36.549814 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.test' with status OK
2026.05.06 08:48:36.549885 [ 31878 ] {} <Debug> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Loaded 5 outdated data parts asynchronously
2026.05.06 08:48:36.549914 [ 32286 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.549946 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.buf' in BackgrndStartup
2026.05.06 08:48:36.549963 [ 32362 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.549940 [ 32384 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.550214 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_m91yt8r2_1.t' with status OK
2026.05.06 08:48:36.550275 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Debug> MergeTask::PrepareStage: Merging 3 parts: from all_1_1_0_6 to all_5_5_0_6 into Wide with storage Full
2026.05.06 08:48:36.550286 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.550406 [ 32286 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.550396 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.data_01279' in BackgrndStartup
2026.05.06 08:48:36.550457 [ 32391 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.550609 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.rename_table_multiple' with status OK
2026.05.06 08:48:36.550777 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.data_01670' in BackgrndStartup
2026.05.06 08:48:36.550906 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.h3_indexes' with status OK
2026.05.06 08:48:36.550913 [ 32393 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.550973 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.mydb' in BackgrndStartup
2026.05.06 08:48:36.551037 [ 32362 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.551101 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.mydb' with status OK
2026.05.06 08:48:36.551145 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.table_for_freeze_old_syntax' in BackgrndStartup
2026.05.06 08:48:36.551188 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_4_4_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.551239 [ 32467 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.table_for_freeze' with status OK
2026.05.06 08:48:36.551361 [ 32467 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.buffer_00763_1' in BackgrndStartup
2026.05.06 08:48:36.551398 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:36.551527 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.buf' with status OK
2026.05.06 08:48:36.551688 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.my_events' in BackgrndStartup
2026.05.06 08:48:36.551747 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0_6, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.551768 [ 32384 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.551789 [ 32284 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part, column a
2026.05.06 08:48:36.551850 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.t_proj_external' with status OK
2026.05.06 08:48:36.551900 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_5_5_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.552158 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.source_table_dictionary' in BackgrndStartup
2026.05.06 08:48:36.552310 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.data_01279' with status OK
2026.05.06 08:48:36.552422 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.table_when_b_even' in BackgrndStartup
2026.05.06 08:48:36.552710 [ 32284 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part, column a
2026.05.06 08:48:36.552881 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0_6, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.553145 [ 32284 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0, total 4 rows starting from the beginning of the part, column a
2026.05.06 08:48:36.553281 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.table_when_b_even' with status OK
2026.05.06 08:48:36.553512 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.input_function_table_2' in BackgrndStartup
2026.05.06 08:48:36.553597 [ 32284 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_4_4_0, total 2 rows starting from the beginning of the part, column a
2026.05.06 08:48:36.553680 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_5_5_0_6, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.553707 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.input_function_table_2' with status OK
2026.05.06 08:48:36.554068 [ 32464 ] {} <Information> AsyncLoader: Processed: 59.5%
2026.05.06 08:48:36.554314 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.t2' in BackgrndStartup
2026.05.06 08:48:36.554426 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 109
2026.05.06 08:48:36.554250 [ 31896 ] {} <Debug> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff) (MergerMutator): Selected 20 parts from all_1_1_0 to all_20_20_0
2026.05.06 08:48:36.554456 [ 32342 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.mv_b_even' with status OK
2026.05.06 08:48:36.554607 [ 32342 ] {} <Debug> AsyncLoader: Execute load job 'startup table 01778_db.hierarchy_source_table' in BackgrndStartup
2026.05.06 08:48:36.554819 [ 32342 ] {} <Debug> AsyncLoader: Finish load job 'startup table 01778_db.hierarchy_source_table' with status OK
2026.05.06 08:48:36.555302 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.555365 [ 32342 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.date_t' in BackgrndStartup
2026.05.06 08:48:36.555436 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 7, bytes: 873
2026.05.06 08:48:36.555444 [ 31896 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:36.555490 [ 31867 ] {} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Creating shared ID for table test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55)
2026.05.06 08:48:36.555503 [ 31856 ] {} <Debug> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Creating shared ID for table shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e)
2026.05.06 08:48:36.555528 [ 31859 ] {} <Debug> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Creating shared ID for table test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708)
2026.05.06 08:48:36.555507 [ 31864 ] {} <Debug> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Creating shared ID for table test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee)
2026.05.06 08:48:36.555516 [ 32467 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.buffer_00763_1' with status OK
2026.05.06 08:48:36.555692 [ 31862 ] {} <Debug> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Creating shared ID for table shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456)
2026.05.06 08:48:36.555730 [ 32467 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.mv_source' in BackgrndStartup
2026.05.06 08:48:36.555729 [ 31858 ] {} <Debug> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Creating shared ID for table test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe)
2026.05.06 08:48:36.555703 [ 31896 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:36.556154 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.source_table_dictionary' with status OK
2026.05.06 08:48:36.556238 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.collapsing_table' in BackgrndStartup
2026.05.06 08:48:36.556383 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.my_events' with status OK
2026.05.06 08:48:36.556545 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.mt' in BackgrndStartup
2026.05.06 08:48:36.556669 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.data_01670' with status OK
2026.05.06 08:48:36.556713 [ 32244 ] {} <Test> KeeperLogStore: Flushing 7 logs
2026.05.06 08:48:36.556738 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.table_with_column_ttl' in BackgrndStartup
2026.05.06 08:48:36.556828 [ 31869 ] {} <Debug> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Creating shared ID for table test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92)
2026.05.06 08:48:36.556914 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.collapsing_table' with status OK
2026.05.06 08:48:36.557312 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 19, bytes: 2011
2026.05.06 08:48:36.557349 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.tab1' in BackgrndStartup
2026.05.06 08:48:36.557318 [ 31904 ] {} <Debug> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Loading 5 outdated data parts asynchronously
2026.05.06 08:48:36.557511 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.table_for_freeze_old_syntax' with status OK
2026.05.06 08:48:36.557620 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTask::PrepareStage: Merging 20 parts: from all_1_1_0 to all_20_20_0 into Compact with storage Full
2026.05.06 08:48:36.557638 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test.avro' in BackgrndStartup
2026.05.06 08:48:36.557731 [ 32435 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Loading Outdated part all_3_3_0 from disk default
2026.05.06 08:48:36.557793 [ 32342 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.date_t' with status OK
2026.05.06 08:48:36.557866 [ 32342 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.set_index_not__fuzz_0' in BackgrndStartup
2026.05.06 08:48:36.557873 [ 32449 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Loading Outdated part all_2_2_0 from disk default
2026.05.06 08:48:36.557998 [ 32458 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Loading Outdated part all_2_2_1 from disk default
2026.05.06 08:48:36.558009 [ 32365 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Loading Outdated part all_1_1_0 from disk default
2026.05.06 08:48:36.558087 [ 32451 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Loading Outdated part all_1_2_2 from disk default
2026.05.06 08:48:36.558258 [ 32342 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.set_index_not__fuzz_0' with status OK
2026.05.06 08:48:36.558277 [ 32244 ] {} <Test> KeeperLogStore: Flushing 19 logs
2026.05.06 08:48:36.558325 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 6, bytes: 783
2026.05.06 08:48:36.558352 [ 32342 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.dist_01781' in BackgrndStartup
2026.05.06 08:48:36.558419 [ 32342 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.dist_01781' with status OK
2026.05.06 08:48:36.558468 [ 32342 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.table_for_freeze_replicated' in BackgrndStartup
2026.05.06 08:48:36.558521 [ 32342 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Starting up table
2026.05.06 08:48:36.558746 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:36.558833 [ 32244 ] {} <Test> KeeperLogStore: Flushing 6 logs
2026.05.06 08:48:36.558927 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.559154 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.t2' with status OK
2026.05.06 08:48:36.559273 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.add_aggregate' in BackgrndStartup
2026.05.06 08:48:36.559345 [ 32391 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.559395 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.mt' with status OK
2026.05.06 08:48:36.559407 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.559463 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table 01945.db.test_dictionary_values' in BackgrndStartup
2026.05.06 08:48:36.559548 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table 01945.db.test_dictionary_values' with status OK
2026.05.06 08:48:36.559599 [ 32472 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.replicated_mutation_table' in BackgrndStartup
2026.05.06 08:48:36.559710 [ 32391 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.559711 [ 32472 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Starting up table
2026.05.06 08:48:36.559751 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test.avro' with status OK
2026.05.06 08:48:36.559765 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.559872 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.view_4' in BackgrndStartup
2026.05.06 08:48:36.559973 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.tab1' with status OK
2026.05.06 08:48:36.560029 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.t' in BackgrndStartup
2026.05.06 08:48:36.560093 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_4_4_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.560091 [ 32362 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.560149 [ 32467 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.mv_source' with status OK
2026.05.06 08:48:36.560231 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 139
2026.05.06 08:48:36.560303 [ 32467 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.t' in BackgrndStartup
2026.05.06 08:48:36.560359 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_5_5_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.560478 [ 32362 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.560507 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.560568 [ 32467 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Starting up table
2026.05.06 08:48:36.560582 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.add_aggregate' with status OK
2026.05.06 08:48:36.560707 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_jbjeaal4_1.t' in BackgrndStartup
2026.05.06 08:48:36.560784 [ 32362 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.560801 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.table_with_column_ttl' with status OK
2026.05.06 08:48:36.560858 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.02952_disjunction_optimization' in BackgrndStartup
2026.05.06 08:48:36.560790 [ 31909 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Loading 2 outdated data parts asynchronously
2026.05.06 08:48:36.560926 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.02952_disjunction_optimization' with status OK
2026.05.06 08:48:36.560967 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.input_function_table_6' in BackgrndStartup
2026.05.06 08:48:36.561012 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.input_function_table_6' with status OK
2026.05.06 08:48:36.561090 [ 32418 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.alter_default' in BackgrndStartup
2026.05.06 08:48:36.561141 [ 32443 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Loading Outdated part all_0_0_0 from disk default
2026.05.06 08:48:36.561163 [ 32418 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Starting up table
2026.05.06 08:48:36.561223 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.t' with status OK
2026.05.06 08:48:36.561260 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.table_src' in BackgrndStartup
2026.05.06 08:48:36.561335 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_jbjeaal4_1.t' with status OK
2026.05.06 08:48:36.561440 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.dynamic_test_2' in BackgrndStartup
2026.05.06 08:48:36.561562 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.dynamic_test_2' with status OK
2026.05.06 08:48:36.561645 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.select_final' in BackgrndStartup
2026.05.06 08:48:36.561739 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_6_6_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.561954 [ 32449 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): loadDataPart: inserting all_2_2_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.561953 [ 31867 ] {} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Initializing table shared ID with 920b4c2c-6de3-4d9a-aa58-0f0e78c16a55
2026.05.06 08:48:36.562048 [ 32449 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Finished loading Outdated part all_2_2_0 on disk default
2026.05.06 08:48:36.562133 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.select_final' with status OK
2026.05.06 08:48:36.562014 [ 31913 ] {} <Debug> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42) (MergerMutator): Selected 8 parts from all_1_1_0 to all_8_8_0
2026.05.06 08:48:36.562185 [ 32449 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938), part: all_2_2_0
2026.05.06 08:48:36.562229 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.mv_target' in BackgrndStartup
2026.05.06 08:48:36.562280 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 145
2026.05.06 08:48:36.562380 [ 32449 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Loading Outdated part all_0_0_1 from disk default
2026.05.06 08:48:36.562408 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.562464 [ 31913 ] {} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:36.562556 [ 31864 ] {} <Debug> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Initializing table shared ID with efd2fe46-3059-4555-b6e3-b045b73e2825
2026.05.06 08:48:36.562669 [ 31862 ] {} <Debug> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Initializing table shared ID with 82bb6fb2-dcb0-4fc4-adbf-fabbfc509456
2026.05.06 08:48:36.562689 [ 32391 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.562711 [ 31913 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:48:36.562733 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.mv_target' with status OK
2026.05.06 08:48:36.562937 [ 31858 ] {} <Debug> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Initializing table shared ID with b7ee17fe-d872-4ddc-8c1c-ede485e51abe
2026.05.06 08:48:36.563025 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.test_s3_mt_dst' in BackgrndStartup
2026.05.06 08:48:36.563118 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.table_src' with status OK
2026.05.06 08:48:36.563168 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.xp' in BackgrndStartup
2026.05.06 08:48:36.563247 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.view_4' with status OK
2026.05.06 08:48:36.563281 [ 32451 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): loadDataPart: inserting all_1_2_2 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.563342 [ 32451 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Finished loading Outdated part all_1_2_2 on disk default
2026.05.06 08:48:36.563346 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.test_table' in BackgrndStartup
2026.05.06 08:48:36.563412 [ 31859 ] {} <Debug> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Initializing table shared ID with b7ee17fe-d872-4ddc-8c1c-ede485e51abe
2026.05.06 08:48:36.563441 [ 32451 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938), part: all_1_2_2
2026.05.06 08:48:36.563430 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.xp' with status OK
2026.05.06 08:48:36.563465 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Debug> MergeTask::PrepareStage: Merging 8 parts: from all_1_1_0 to all_8_8_0 into Compact with storage Full
2026.05.06 08:48:36.563859 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table rdb_test_13.t3' in BackgrndStartup
2026.05.06 08:48:36.563895 [ 32435 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): loadDataPart: inserting all_3_3_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.563926 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_7_7_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.563988 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.test_table' with status OK
2026.05.06 08:48:36.564022 [ 32435 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Finished loading Outdated part all_3_3_0 on disk default
2026.05.06 08:48:36.564063 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.t1' in BackgrndStartup
2026.05.06 08:48:36.564105 [ 32458 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): loadDataPart: inserting all_2_2_1 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.564156 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table rdb_test_13.t3' with status OK
2026.05.06 08:48:36.564206 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:36.564272 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.set' in BackgrndStartup
2026.05.06 08:48:36.564280 [ 32435 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938), part: all_3_3_0
2026.05.06 08:48:36.564364 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_8_8_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.564356 [ 31869 ] {} <Debug> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Initializing table shared ID with c8e903e3-5254-4279-92c8-4041d14dee92
2026.05.06 08:48:36.564294 [ 32458 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Finished loading Outdated part all_2_2_1 on disk default
2026.05.06 08:48:36.564382 [ 32384 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.564453 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.set' with status OK
2026.05.06 08:48:36.564461 [ 32365 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): loadDataPart: inserting all_1_1_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.564500 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.564564 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.input_function_table_5' in BackgrndStartup
2026.05.06 08:48:36.564572 [ 31856 ] {} <Debug> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Initializing table shared ID with 82bb6fb2-dcb0-4fc4-adbf-fabbfc509456
2026.05.06 08:48:36.564610 [ 32365 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Finished loading Outdated part all_1_1_0 on disk default
2026.05.06 08:48:36.564645 [ 32458 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938), part: all_2_2_1
2026.05.06 08:48:36.564689 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_9_9_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.564733 [ 32362 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.564660 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.input_function_table_5' with status OK
2026.05.06 08:48:36.564825 [ 32365 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938), part: all_1_1_0
2026.05.06 08:48:36.564869 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.564809 [ 32384 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.565054 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_10_10_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.565098 [ 32391 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.565338 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_11_11_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.565362 [ 31904 ] {} <Debug> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Loaded 5 outdated data parts asynchronously
2026.05.06 08:48:36.565388 [ 32384 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.565425 [ 31919 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:36.565411 [ 32393 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.565673 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_12_12_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.565693 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.565700 [ 32362 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.566060 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_13_13_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.566123 [ 32461 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.566175 [ 32393 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.566340 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_4_4_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.566420 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 34, bytes: 4146
2026.05.06 08:48:36.566525 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_14_14_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.566582 [ 32391 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.566798 [ 32449 ] {} <Test> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): loadDataPart: inserting all_0_0_1 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.566811 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.input_function_table_4' in BackgrndStartup
2026.05.06 08:48:36.566858 [ 32449 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Finished loading Outdated part all_0_0_1 on disk default
2026.05.06 08:48:36.566903 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_15_15_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.566924 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.t1' with status OK
2026.05.06 08:48:36.566940 [ 32362 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.566933 [ 32384 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.566950 [ 32449 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd), part: all_0_0_1
2026.05.06 08:48:36.567015 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.x' in BackgrndStartup
2026.05.06 08:48:36.567219 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_16_16_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.567261 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_5_5_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.567305 [ 32391 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.567368 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.input_function_table_4' with status OK
2026.05.06 08:48:36.567509 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_17_17_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.567649 [ 32384 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.567760 [ 32457 ] {} <Debug> AsyncLoader: Execute load job 'startup table shard_1.from_1' in BackgrndStartup
2026.05.06 08:48:36.567869 [ 32457 ] {} <Trace> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Starting up table
2026.05.06 08:48:36.567908 [ 31855 ] {} <Debug> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Creating shared ID for table test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d)
2026.05.06 08:48:36.567927 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.x' with status OK
2026.05.06 08:48:36.568089 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_xfltqef0.mt' in BackgrndStartup
2026.05.06 08:48:36.568156 [ 32384 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.568661 [ 32384 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.568695 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 29, bytes: 3519
2026.05.06 08:48:36.568705 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_xfltqef0.mt' with status OK
2026.05.06 08:48:36.568663 [ 32244 ] {} <Test> KeeperLogStore: Flushing 34 logs
2026.05.06 08:48:36.568880 [ 32446 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.insert_dedup_token2' in BackgrndStartup
2026.05.06 08:48:36.568982 [ 32446 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Starting up table
2026.05.06 08:48:36.568678 [ 32443 ] {} <Test> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): loadDataPart: inserting all_0_0_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.569682 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_18_18_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.569768 [ 32443 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Finished loading Outdated part all_0_0_0 on disk default
2026.05.06 08:48:36.569866 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_6_6_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.569971 [ 32443 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd), part: all_0_0_0
2026.05.06 08:48:36.570022 [ 32391 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.570049 [ 32276 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_19_19_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.570172 [ 31909 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Loaded 2 outdated data parts asynchronously
2026.05.06 08:48:36.570202 [ 32244 ] {} <Test> KeeperLogStore: Flushing 29 logs
2026.05.06 08:48:36.570228 [ 32464 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:36.570578 [ 32384 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.570587 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_7_7_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.570807 [ 32440 ] {696a4bd3-28ea-4388-b32e-930c9288432f} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:36.571047 [ 32464 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:36 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE72916F5BE5B
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:36.571095 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_8_8_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.571116 [ 32461 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.571226 [ 32440 ] {696a4bd3-28ea-4388-b32e-930c9288432f} <Trace> AggregatingTransform: Aggregated. 0 to 1 rows (from 0.00 B) in 0.013592797 sec. (0.000 rows/sec., 0.00 B/sec.)
2026.05.06 08:48:36.571723 [ 32393 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.571652 [ 32440 ] {696a4bd3-28ea-4388-b32e-930c9288432f} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:36.572014 [ 32464 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:36.572276 [ 32440 ] {696a4bd3-28ea-4388-b32e-930c9288432f} <Trace> HashTablesStatistics: Statistics updated for key=16690531032610678897: new sum_of_sizes=1, median_size=1
2026.05.06 08:48:36.573005 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.test_s3_mt_dst' with status OK
2026.05.06 08:48:36.573078 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.data2013' in BackgrndStartup
2026.05.06 08:48:36.573139 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.data2013' with status OK
2026.05.06 08:48:36.573174 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_3.ordinary_distinct' in BackgrndStartup
2026.05.06 08:48:36.573446 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_3.ordinary_distinct' with status OK
2026.05.06 08:48:36.573503 [ 32464 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.rmt1' in BackgrndStartup
2026.05.06 08:48:36.573640 [ 32464 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Starting up table
2026.05.06 08:48:36.573826 [ 31926 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Loading 2 unexpected data parts
2026.05.06 08:48:36.574859 [ 32475 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Loading unexpected part all_1_1_0_2 from disk s3_disk
2026.05.06 08:48:36.574843 [ 32428 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Loading unexpected part all_0_0_0_2 from disk s3_disk
2026.05.06 08:48:36.575013 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 17, bytes: 2281
2026.05.06 08:48:36.576452 [ 31884 ] {} <Debug> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Creating shared ID for table test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67)
2026.05.06 08:48:36.576816 [ 32475 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/count.txt)
2026.05.06 08:48:36.576815 [ 32428 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/count.txt)
2026.05.06 08:48:36.577278 [ 32475 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:36.577338 [ 32244 ] {} <Test> KeeperLogStore: Flushing 17 logs
2026.05.06 08:48:36.577279 [ 32428 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:36.577555 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 7, bytes: 1121
2026.05.06 08:48:36.577551 [ 32286 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 3 rows, containing 2 columns (2 merged, 0 gathered) in 0.035839895 sec., 83.70560237411411 rows/sec., 1.47 KiB/sec.
2026.05.06 08:48:36.577753 [ 32428 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:36.577753 [ 32475 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:36.578390 [ 31855 ] {} <Debug> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Initializing table shared ID with 91844baf-1583-4a3f-9f40-e6d36837eb7d
2026.05.06 08:48:36.578383 [ 32475 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:UnexpectedParts:32475, kind: Regular, unbound: 0
2026.05.06 08:48:36.578384 [ 32428 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:UnexpectedParts:32428, kind: Regular, unbound: 0
2026.05.06 08:48:36.578925 [ 32286 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Trace> MergedBlockOutputStream: filled checksums all_1_3_1 (state Temporary)
2026.05.06 08:48:36.579020 [ 32428 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:UnexpectedParts:32428, kind: Regular, unbound: 0
2026.05.06 08:48:36.579020 [ 32475 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:UnexpectedParts:32475, kind: Regular, unbound: 0
2026.05.06 08:48:36.579535 [ 32428 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:36.579535 [ 32475 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:36.579826 [ 32428 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:36.579827 [ 32475 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:36.580027 [ 31919 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:48:36.580072 [ 32244 ] {} <Test> KeeperLogStore: Flushing 7 logs
2026.05.06 08:48:36.580351 [ 32428 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:36.580352 [ 32475 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:36.581159 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 9, bytes: 1338
2026.05.06 08:48:36.581210 [ 32275 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Trace> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): Renaming temporary part tmp_merge_all_1_3_1 to all_1_3_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:36.581161 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part, column b
2026.05.06 08:48:36.581374 [ 31884 ] {} <Debug> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Initializing table shared ID with 608153d5-71f1-4052-9b53-8112203d1f67
2026.05.06 08:48:36.581604 [ 32275 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Test> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b): preparePartForCommit: inserting all_1_3_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:36.581603 [ 31926 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Loaded 2 unexpected data parts
2026.05.06 08:48:36.581742 [ 32275 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Trace> test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b) (MergerMutator): Merged 3 parts: [all_1_1_0, all_3_3_0] -> all_1_3_1
2026.05.06 08:48:36.582008 [ 32275 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b), part: all_1_1_0
2026.05.06 08:48:36.582164 [ 32275 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b), part: all_2_2_0
2026.05.06 08:48:36.582293 [ 32275 ] {64b53cb6-662a-4a2f-8d57-30d7247e140b::all_1_3_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_13.array_jaccard_index (64b53cb6-662a-4a2f-8d57-30d7247e140b), part: all_3_3_0
2026.05.06 08:48:36.582345 [ 32244 ] {} <Test> KeeperLogStore: Flushing 9 logs
2026.05.06 08:48:36.582784 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part, column b
2026.05.06 08:48:36.583152 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0, total 4 rows starting from the beginning of the part, column b
2026.05.06 08:48:36.583249 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 8 rows, containing 3 columns (3 merged, 0 gathered) in 0.019994775 sec., 400.10452730775916 rows/sec., 11.72 KiB/sec.
2026.05.06 08:48:36.583645 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_4_4_0, total 2 rows starting from the beginning of the part, column b
2026.05.06 08:48:36.583966 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 9, bytes: 1599
2026.05.06 08:48:36.584427 [ 31862 ] {} <Debug> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Creating shared ID for table shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456)
2026.05.06 08:48:36.584427 [ 31864 ] {} <Debug> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Creating shared ID for table test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee)
2026.05.06 08:48:36.584432 [ 31858 ] {} <Debug> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Creating shared ID for table test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe)
2026.05.06 08:48:36.584762 [ 31869 ] {} <Debug> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Creating shared ID for table test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92)
2026.05.06 08:48:36.584800 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Trace> MergedBlockOutputStream: filled checksums all_1_8_1 (state Temporary)
2026.05.06 08:48:36.584427 [ 31867 ] {} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Creating shared ID for table test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55)
2026.05.06 08:48:36.584467 [ 31859 ] {} <Debug> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Creating shared ID for table test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708)
2026.05.06 08:48:36.584650 [ 31856 ] {} <Debug> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Creating shared ID for table shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e)
2026.05.06 08:48:36.585217 [ 32244 ] {} <Test> KeeperLogStore: Flushing 9 logs
2026.05.06 08:48:36.585302 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 2, bytes: 315
2026.05.06 08:48:36.585408 [ 31858 ] {} <Information> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Shared ID already set to b7ee17fe-d872-4ddc-8c1c-ede485e51abe
2026.05.06 08:48:36.585415 [ 31869 ] {} <Information> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Shared ID already set to c8e903e3-5254-4279-92c8-4041d14dee92
2026.05.06 08:48:36.585452 [ 31864 ] {} <Information> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Shared ID already set to efd2fe46-3059-4555-b6e3-b045b73e2825
2026.05.06 08:48:36.585407 [ 31862 ] {} <Information> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Shared ID already set to 82bb6fb2-dcb0-4fc4-adbf-fabbfc509456
2026.05.06 08:48:36.585415 [ 31867 ] {} <Information> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Shared ID already set to 920b4c2c-6de3-4d9a-aa58-0f0e78c16a55
2026.05.06 08:48:36.585599 [ 31859 ] {} <Information> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Shared ID already set to b7ee17fe-d872-4ddc-8c1c-ede485e51abe
2026.05.06 08:48:36.585604 [ 31641 ] {696a4bd3-28ea-4388-b32e-930c9288432f} <Debug> executeQuery: Read 66 rows, 4.97 KiB in 0.059167 sec., 1115.4866733145166 rows/sec., 83.98 KiB/sec.
2026.05.06 08:48:36.585907 [ 31856 ] {} <Information> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Shared ID already set to 82bb6fb2-dcb0-4fc4-adbf-fabbfc509456
2026.05.06 08:48:36.586443 [ 32244 ] {} <Test> KeeperLogStore: Flushing 2 logs
2026.05.06 08:48:36.586588 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 4, bytes: 420
2026.05.06 08:48:36.586951 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 3 rows, containing 4 columns (4 merged, 0 gathered) in 0.037059807 sec., 80.95023268739635 rows/sec., 2.37 KiB/sec.
2026.05.06 08:48:36.587189 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): Renaming temporary part tmp_merge_all_1_8_1 to all_1_8_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:36.587668 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42): preparePartForCommit: inserting all_1_8_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:36.588445 [ 32244 ] {} <Test> KeeperLogStore: Flushing 4 logs
2026.05.06 08:48:36.588644 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Trace> test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42) (MergerMutator): Merged 8 parts: [all_1_1_0, all_8_8_0] -> all_1_8_1
2026.05.06 08:48:36.588957 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42), part: all_1_1_0
2026.05.06 08:48:36.589002 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 2, bytes: 373
2026.05.06 08:48:36.589178 [ 32244 ] {} <Test> KeeperLogStore: Flushing 2 logs
2026.05.06 08:48:36.589198 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42), part: all_2_2_0
2026.05.06 08:48:36.589303 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42), part: all_3_3_0
2026.05.06 08:48:36.589329 [ 32275 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 4.12 MiB.
2026.05.06 08:48:36.589371 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42), part: all_4_4_0
2026.05.06 08:48:36.589437 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42), part: all_5_5_0
2026.05.06 08:48:36.589510 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42), part: all_6_6_0
2026.05.06 08:48:36.589573 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42), part: all_7_7_0
2026.05.06 08:48:36.589636 [ 32285 ] {3ede77ca-24a6-48a6-8fb8-934575532f42::all_1_8_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_src (3ede77ca-24a6-48a6-8fb8-934575532f42), part: all_8_8_0
2026.05.06 08:48:36.589983 [ 31862 ] {} <Information> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Became leader
2026.05.06 08:48:36.589984 [ 31867 ] {} <Information> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Became leader
2026.05.06 08:48:36.589984 [ 31869 ] {} <Information> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Became leader
2026.05.06 08:48:36.589984 [ 31864 ] {} <Information> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Became leader
2026.05.06 08:48:36.589985 [ 31856 ] {} <Information> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Became leader
2026.05.06 08:48:36.590230 [ 31867 ] {} <Trace> test_1.test_without_merge (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.590382 [ 31858 ] {} <Information> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Became leader
2026.05.06 08:48:36.590231 [ 31869 ] {} <Trace> test_15.src (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.590296 [ 31859 ] {} <Information> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Became leader
2026.05.06 08:48:36.590460 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 83
2026.05.06 08:48:36.590393 [ 31864 ] {} <Trace> test_15.partslost_1 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.590469 [ 31858 ] {} <Trace> test_15.t (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.590558 [ 31864 ] {} <Trace> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Trying to startup table from right now
2026.05.06 08:48:36.590695 [ 31855 ] {} <Debug> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Creating shared ID for table test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d)
2026.05.06 08:48:36.590489 [ 31867 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Trying to startup table from right now
2026.05.06 08:48:36.590480 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Trace> MergedBlockOutputStream: filled checksums all_1_5_1_6 (state Temporary)
2026.05.06 08:48:36.590461 [ 31856 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.590489 [ 31869 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Trying to startup table from right now
2026.05.06 08:48:36.590563 [ 31859 ] {} <Trace> test_15.t_r_ok (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.590230 [ 31862 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.590600 [ 31858 ] {} <Trace> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Trying to startup table from right now
2026.05.06 08:48:36.590745 [ 31855 ] {} <Information> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Shared ID already set to 91844baf-1583-4a3f-9f40-e6d36837eb7d
2026.05.06 08:48:36.590757 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.591246 [ 31859 ] {} <Trace> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Trying to startup table from right now
2026.05.06 08:48:36.591265 [ 31856 ] {} <Trace> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Trying to startup table from right now
2026.05.06 08:48:36.591276 [ 31867 ] {} <Debug> test_1.test_without_merge (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.591325 [ 31869 ] {} <Debug> test_15.src (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.591277 [ 31864 ] {} <Debug> test_15.partslost_1 (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.591344 [ 31859 ] {} <Debug> test_15.t_r_ok (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.591520 [ 31862 ] {} <Trace> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Trying to startup table from right now
2026.05.06 08:48:36.591372 [ 31858 ] {} <Debug> test_15.t (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.591588 [ 31862 ] {} <Debug> shard_0.from_0 (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.591379 [ 31856 ] {} <Debug> shard_1.from_0 (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.591795 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 67
2026.05.06 08:48:36.592169 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.592552 [ 31884 ] {} <Debug> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Creating shared ID for table test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67)
2026.05.06 08:48:36.592791 [ 31856 ] {} <Debug> shard_1.from_0 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.592789 [ 31867 ] {} <Debug> test_1.test_without_merge (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.592793 [ 31858 ] {} <Debug> test_15.t (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.593024 [ 31884 ] {} <Information> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Shared ID already set to 608153d5-71f1-4052-9b53-8112203d1f67
2026.05.06 08:48:36.592788 [ 31862 ] {} <Debug> shard_0.from_0 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.592790 [ 31869 ] {} <Debug> test_15.src (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.592793 [ 31859 ] {} <Debug> test_15.t_r_ok (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.592795 [ 31864 ] {} <Debug> test_15.partslost_1 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.593580 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 89
2026.05.06 08:48:36.594273 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.595546 [ 32285 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 4.11 MiB.
2026.05.06 08:48:36.595580 [ 31641 ] {696a4bd3-28ea-4388-b32e-930c9288432f} <Debug> TCPHandler: Processed in 0.07210367 sec.
2026.05.06 08:48:36.598687 [ 31855 ] {} <Information> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Became leader
2026.05.06 08:48:36.598794 [ 31855 ] {} <Trace> test_15.dst (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.598869 [ 31855 ] {} <Trace> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Trying to startup table from right now
2026.05.06 08:48:36.598932 [ 31855 ] {} <Debug> test_15.dst (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.599004 [ 31855 ] {} <Debug> test_15.dst (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.599393 [ 31884 ] {} <Information> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Became leader
2026.05.06 08:48:36.599512 [ 31884 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.599584 [ 31884 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Trying to startup table from right now
2026.05.06 08:48:36.600498 [ 31884 ] {} <Debug> test_13.table_for_rename_replicated (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.600680 [ 31884 ] {} <Debug> test_13.table_for_rename_replicated (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.602973 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 4, bytes: 270
2026.05.06 08:48:36.604214 [ 31912 ] {} <Debug> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Creating shared ID for table test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3)
2026.05.06 08:48:36.605062 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Trace> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): Renaming temporary part tmp_merge_all_1_5_1_6 to all_1_5_1_6 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:36.605244 [ 32244 ] {} <Test> KeeperLogStore: Flushing 4 logs
2026.05.06 08:48:36.605426 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Test> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92): preparePartForCommit: inserting all_1_5_1_6 (state PreActive) into data_parts_indexes
2026.05.06 08:48:36.605583 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Trace> test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92) (MergerMutator): Merged 3 parts: [all_1_1_0_6, all_5_5_0_6] -> all_1_5_1_6
2026.05.06 08:48:36.605880 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92), part: all_1_1_0_6
2026.05.06 08:48:36.605947 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 9, bytes: 1344
2026.05.06 08:48:36.606044 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92), part: all_3_3_0_6
2026.05.06 08:48:36.606196 [ 32280 ] {c691d315-a30a-48aa-abb7-5b1d8f0afa92::all_1_5_1_6} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_13.rename_table_multiple (c691d315-a30a-48aa-abb7-5b1d8f0afa92), part: all_5_5_0_6
2026.05.06 08:48:36.607167 [ 32280 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 8.30 MiB.
2026.05.06 08:48:36.607304 [ 32244 ] {} <Test> KeeperLogStore: Flushing 9 logs
2026.05.06 08:48:36.608376 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 5, bytes: 1419
2026.05.06 08:48:36.608691 [ 31912 ] {} <Debug> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Initializing table shared ID with 35bad05d-d8fe-4647-a804-b081c3be65d3
2026.05.06 08:48:36.610135 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part, column _block_offset
2026.05.06 08:48:36.610258 [ 32244 ] {} <Test> KeeperLogStore: Flushing 5 logs
2026.05.06 08:48:36.610380 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 12, bytes: 2460
2026.05.06 08:48:36.610580 [ 31911 ] {} <Debug> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Creating shared ID for table test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1)
2026.05.06 08:48:36.610602 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part, column _block_offset
2026.05.06 08:48:36.610818 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0, total 4 rows starting from the beginning of the part, column _block_offset
2026.05.06 08:48:36.610992 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_4_4_0, total 2 rows starting from the beginning of the part, column _block_offset
2026.05.06 08:48:36.611847 [ 32244 ] {} <Test> KeeperLogStore: Flushing 12 logs
2026.05.06 08:48:36.611900 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 5, bytes: 640
2026.05.06 08:48:36.612754 [ 32244 ] {} <Test> KeeperLogStore: Flushing 5 logs
2026.05.06 08:48:36.612789 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 457
2026.05.06 08:48:36.613091 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.613158 [ 31927 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Uncovered unexpected parts: . Restorable unexpected parts: all_1_1_0_2, all_0_0_0_2. Missing parts: . Covered unexpected parts: . Expected parts: all_1_1_0, all_0_0_0.
2026.05.06 08:48:36.613625 [ 31927 ] {} <Warning> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): The local set of parts of table test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) doesn't look like the set of parts in ZooKeeper: 0.00 rows of 4.00 total rows in filesystem are suspicious. There are 0 uncovered unexpected parts with 0 rows (0 of them is not just-written with 0 rows), 0 missing parts (with 0 blocks), 0 covered unexpected parts (with 2 rows).
2026.05.06 08:48:36.613746 [ 31864 ] {} <Trace> test_15.partslost_1 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.613932 [ 31921 ] {} <Debug> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Creating shared ID for table shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826)
2026.05.06 08:48:36.614002 [ 31911 ] {} <Debug> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Initializing table shared ID with 9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1
2026.05.06 08:48:36.614055 [ 31923 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Creating shared ID for table test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355)
2026.05.06 08:48:36.614104 [ 31867 ] {} <Trace> test_1.test_without_merge (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.614145 [ 31855 ] {} <Trace> test_15.dst (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.614234 [ 31859 ] {} <Trace> test_15.t_r_ok (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.614273 [ 31858 ] {} <Trace> test_15.t (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.614919 [ 31927 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Creating shared ID for table test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845)
2026.05.06 08:48:36.614936 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:36.615121 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 17, bytes: 2179
2026.05.06 08:48:36.615333 [ 31869 ] {} <Trace> test_15.src (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.615426 [ 31862 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.615473 [ 31856 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.616200 [ 32244 ] {} <Test> KeeperLogStore: Flushing 17 logs
2026.05.06 08:48:36.616977 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 9, bytes: 1046
2026.05.06 08:48:36.617406 [ 31921 ] {} <Debug> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Initializing table shared ID with 703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7
2026.05.06 08:48:36.617588 [ 31906 ] {} <Debug> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Creating shared ID for table test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a)
2026.05.06 08:48:36.617708 [ 32244 ] {} <Test> KeeperLogStore: Flushing 9 logs
2026.05.06 08:48:36.617627 [ 31864 ] {} <Test> test_15.partslost_1 (ReplicatedMergeTreeQueue): Adding part all_0_0_0_2 to current and virtual parts
2026.05.06 08:48:36.618104 [ 31923 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Initializing table shared ID with 0d2b666f-a2a0-48ee-95c0-9c76f867f0f1
2026.05.06 08:48:36.618254 [ 31867 ] {} <Test> test_1.test_without_merge (ReplicatedMergeTreeQueue): Adding part all_2_2_0 to current and virtual parts
2026.05.06 08:48:36.618346 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 4, bytes: 634
2026.05.06 08:48:36.618400 [ 31855 ] {} <Trace> test_15.dst (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.618439 [ 31867 ] {} <Test> test_1.test_without_merge (ReplicatedMergeTreeQueue): Adding part all_0_0_0 to current and virtual parts
2026.05.06 08:48:36.618438 [ 31864 ] {} <Trace> test_15.partslost_1 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.618526 [ 31867 ] {} <Test> test_1.test_without_merge (ReplicatedMergeTreeQueue): Adding part all_3_3_0 to current and virtual parts
2026.05.06 08:48:36.618595 [ 31867 ] {} <Trace> test_1.test_without_merge (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.618644 [ 31884 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.618644 [ 31859 ] {} <Trace> test_15.t_r_ok (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.618696 [ 32244 ] {} <Test> KeeperLogStore: Flushing 4 logs
2026.05.06 08:48:36.618717 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 161
2026.05.06 08:48:36.618659 [ 31858 ] {} <Trace> test_15.t (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.618888 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.619140 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 115
2026.05.06 08:48:36.619188 [ 31927 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Initializing table shared ID with 2edd1075-136c-4caa-a6fc-8fb7a8cd3845
2026.05.06 08:48:36.619226 [ 31869 ] {} <Test> test_15.src (ReplicatedMergeTreeQueue): Adding part 0_0_0_0 to current and virtual parts
2026.05.06 08:48:36.619281 [ 31862 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.619319 [ 31869 ] {} <Test> test_15.src (ReplicatedMergeTreeQueue): Adding part 1_3_3_0 to current and virtual parts
2026.05.06 08:48:36.619358 [ 31869 ] {} <Test> test_15.src (ReplicatedMergeTreeQueue): Adding part 2_0_0_0 to current and virtual parts
2026.05.06 08:48:36.619331 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:36.618820 [ 31864 ] {} <Debug> test_15.partslost_1 (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/table/test_15_02067_lost/partslost/replicas/1/queue
2026.05.06 08:48:36.619391 [ 31856 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.619478 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 7, bytes: 862
2026.05.06 08:48:36.619495 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.619523 [ 31912 ] {} <Debug> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Creating shared ID for table test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3)
2026.05.06 08:48:36.618820 [ 31867 ] {} <Debug> test_1.test_without_merge (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/test_1/test02676_without_merge/replicas/node/queue
2026.05.06 08:48:36.618887 [ 31858 ] {} <Debug> test_15.t (ReplicatedMergeTreeQueue): Loading queue from /tables/test_15/t/replicas/r1/queue
2026.05.06 08:48:36.618820 [ 31855 ] {} <Debug> test_15.dst (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/01034_move_partition_from_table_zookeeper_test_15/dst2/replicas/1/queue
2026.05.06 08:48:36.618819 [ 31859 ] {} <Debug> test_15.t_r_ok (ReplicatedMergeTreeQueue): Loading queue from /tables/test_15/t/replicas/r2/queue
2026.05.06 08:48:36.619388 [ 31869 ] {} <Test> test_15.src (ReplicatedMergeTreeQueue): Adding part 1_4_4_0 to current and virtual parts
2026.05.06 08:48:36.619777 [ 31862 ] {} <Debug> shard_0.from_0 (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/from_0_test_uasu1ifx/replicas/0/queue
2026.05.06 08:48:36.619892 [ 31869 ] {} <Trace> test_15.src (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.619910 [ 31856 ] {} <Debug> shard_1.from_0 (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/from_0_test_uasu1ifx/replicas/1/queue
2026.05.06 08:48:36.619989 [ 31869 ] {} <Debug> test_15.src (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/01034_move_partition_from_table_zookeeper_test_15/src1/replicas/1/queue
2026.05.06 08:48:36.620042 [ 31912 ] {} <Information> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Shared ID already set to 35bad05d-d8fe-4647-a804-b081c3be65d3
2026.05.06 08:48:36.620495 [ 32244 ] {} <Test> KeeperLogStore: Flushing 7 logs
2026.05.06 08:48:36.620549 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 12, bytes: 1543
2026.05.06 08:48:36.620557 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 8 rows, containing 3 columns (1 merged, 2 gathered) in 0.081128872 sec., 98.60854468677933 rows/sec., 1.73 KiB/sec.
2026.05.06 08:48:36.621162 [ 31906 ] {} <Debug> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Initializing table shared ID with 2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a
2026.05.06 08:48:36.621404 [ 32244 ] {} <Test> KeeperLogStore: Flushing 12 logs
2026.05.06 08:48:36.621548 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Trace> MergedBlockOutputStream: filled checksums all_1_4_1 (state Temporary)
2026.05.06 08:48:36.622413 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 13, bytes: 1340
2026.05.06 08:48:36.622805 [ 31884 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Adding part 20191002_0_0_0_1 to current and virtual parts
2026.05.06 08:48:36.622850 [ 31884 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Adding part 20191003_0_0_0_1 to current and virtual parts
2026.05.06 08:48:36.622880 [ 31884 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Adding part 20191001_0_0_0_1 to current and virtual parts
2026.05.06 08:48:36.622913 [ 31884 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.622995 [ 31884 ] {} <Debug> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/01213_alter_rename_column_zookeeper_long_test_13/table_for_rename_replicated/replicas/1/queue
2026.05.06 08:48:36.623093 [ 32244 ] {} <Test> KeeperLogStore: Flushing 13 logs
2026.05.06 08:48:36.624336 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 13, bytes: 2014
2026.05.06 08:48:36.625294 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): Renaming temporary part tmp_merge_all_1_4_1 to all_1_4_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:36.625533 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Test> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95): preparePartForCommit: inserting all_1_4_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:36.625551 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 115
2026.05.06 08:48:36.625533 [ 32244 ] {} <Test> KeeperLogStore: Flushing 13 logs
2026.05.06 08:48:36.625622 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Trace> test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95) (MergerMutator): Merged 4 parts: [all_1_1_0, all_4_4_0] -> all_1_4_1
2026.05.06 08:48:36.625751 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.625782 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95), part: all_1_1_0
2026.05.06 08:48:36.625865 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95), part: all_2_2_0
2026.05.06 08:48:36.625921 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95), part: all_3_3_0
2026.05.06 08:48:36.625983 [ 32283 ] {78e1a3a0-a278-42d4-a1dc-29f769bc1c95::all_1_4_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_w50hoezt.test_constraints (78e1a3a0-a278-42d4-a1dc-29f769bc1c95), part: all_4_4_0
2026.05.06 08:48:36.625136 [ 31912 ] {} <Information> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Became leader
2026.05.06 08:48:36.626255 [ 31912 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.626298 [ 31869 ] {} <Debug> test_15.src (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.626337 [ 31912 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Trying to startup table from right now
2026.05.06 08:48:36.626299 [ 31867 ] {} <Debug> test_1.test_without_merge (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.626301 [ 31864 ] {} <Debug> test_15.partslost_1 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.626301 [ 31862 ] {} <Debug> shard_0.from_0 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.626301 [ 31858 ] {} <Debug> test_15.t (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.626519 [ 31912 ] {} <Debug> test_1.replicated_mutation_table (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.626319 [ 31859 ] {} <Debug> test_15.t_r_ok (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.626336 [ 31856 ] {} <Debug> shard_1.from_0 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.626300 [ 31855 ] {} <Debug> test_15.dst (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.626565 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 114
2026.05.06 08:48:36.626910 [ 31912 ] {} <Debug> test_1.replicated_mutation_table (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.627032 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.627889 [ 32283 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 10.32 MiB.
2026.05.06 08:48:36.628047 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 174
2026.05.06 08:48:36.628314 [ 31911 ] {} <Debug> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Creating shared ID for table test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1)
2026.05.06 08:48:36.628378 [ 31911 ] {} <Information> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Shared ID already set to 9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1
2026.05.06 08:48:36.628406 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.628453 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 173
2026.05.06 08:48:36.628416 [ 31884 ] {} <Debug> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.628756 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.628776 [ 31869 ] {} <Debug> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Updating strategy picker state
2026.05.06 08:48:36.628796 [ 31859 ] {} <Debug> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Updating strategy picker state
2026.05.06 08:48:36.628778 [ 31867 ] {} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Updating strategy picker state
2026.05.06 08:48:36.628826 [ 31858 ] {} <Debug> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Updating strategy picker state
2026.05.06 08:48:36.628937 [ 31864 ] {} <Debug> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Updating strategy picker state
2026.05.06 08:48:36.628982 [ 31862 ] {} <Debug> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Updating strategy picker state
2026.05.06 08:48:36.629026 [ 31856 ] {} <Debug> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Updating strategy picker state
2026.05.06 08:48:36.629086 [ 31855 ] {} <Debug> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Updating strategy picker state
2026.05.06 08:48:36.629393 [ 31921 ] {} <Debug> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Creating shared ID for table shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826)
2026.05.06 08:48:36.629510 [ 31921 ] {} <Information> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Shared ID already set to 703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7
2026.05.06 08:48:36.629516 [ 31923 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Creating shared ID for table test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355)
2026.05.06 08:48:36.629572 [ 31911 ] {} <Information> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Became leader
2026.05.06 08:48:36.629631 [ 31923 ] {} <Information> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Shared ID already set to 0d2b666f-a2a0-48ee-95c0-9c76f867f0f1
2026.05.06 08:48:36.629656 [ 31927 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Creating shared ID for table test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845)
2026.05.06 08:48:36.629661 [ 31911 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.629772 [ 31927 ] {} <Information> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Shared ID already set to 2edd1075-136c-4caa-a6fc-8fb7a8cd3845
2026.05.06 08:48:36.629751 [ 31884 ] {} <Debug> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Updating strategy picker state
2026.05.06 08:48:36.629808 [ 31911 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Trying to startup table from right now
2026.05.06 08:48:36.629843 [ 31911 ] {} <Debug> test_13.alter_default (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.629877 [ 31911 ] {} <Debug> test_13.alter_default (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.630106 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 418
2026.05.06 08:48:36.630544 [ 31906 ] {} <Debug> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Creating shared ID for table test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a)
2026.05.06 08:48:36.630612 [ 31921 ] {} <Information> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Became leader
2026.05.06 08:48:36.630619 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.630643 [ 31906 ] {} <Information> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Shared ID already set to 2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a
2026.05.06 08:48:36.630677 [ 31923 ] {} <Information> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Became leader
2026.05.06 08:48:36.630699 [ 31921 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.630862 [ 31921 ] {} <Trace> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Trying to startup table from right now
2026.05.06 08:48:36.630892 [ 31923 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.630921 [ 31921 ] {} <Debug> shard_1.from_1 (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.630956 [ 31923 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Trying to startup table from right now
2026.05.06 08:48:36.630990 [ 31921 ] {} <Debug> shard_1.from_1 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.631012 [ 31923 ] {} <Debug> test_1.insert_dedup_token2 (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.631085 [ 31923 ] {} <Debug> test_1.insert_dedup_token2 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.631762 [ 31927 ] {} <Information> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Became leader
2026.05.06 08:48:36.631847 [ 31927 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.631916 [ 31927 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to startup table from right now
2026.05.06 08:48:36.631951 [ 31927 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.631953 [ 31867 ] {} <Trace> test_1.test_without_merge (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.631991 [ 31927 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.632083 [ 31864 ] {} <Trace> test_15.partslost_1 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.632266 [ 31869 ] {} <Trace> test_15.src (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.632401 [ 31855 ] {} <Trace> test_15.dst (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.632470 [ 31906 ] {} <Information> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Became leader
2026.05.06 08:48:36.632597 [ 31906 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.632756 [ 31912 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.632778 [ 31906 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Trying to startup table from right now
2026.05.06 08:48:36.632850 [ 31884 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.632859 [ 31906 ] {} <Debug> test_13.table_for_freeze_replicated (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.632994 [ 31906 ] {} <Debug> test_13.table_for_freeze_replicated (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.633086 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 80
2026.05.06 08:48:36.633201 [ 31858 ] {} <Debug> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Strategy picker state updated, current replica: 0, active replicas: [r1, r2]
2026.05.06 08:48:36.633254 [ 31862 ] {} <Debug> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Strategy picker state updated, current replica: 0, active replicas: [0, 1]
2026.05.06 08:48:36.633318 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.633453 [ 31862 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.633456 [ 31859 ] {} <Debug> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Strategy picker state updated, current replica: 1, active replicas: [r1, r2]
2026.05.06 08:48:36.633557 [ 31859 ] {} <Trace> test_15.t_r_ok (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.633595 [ 31856 ] {} <Debug> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Strategy picker state updated, current replica: 1, active replicas: [0, 1]
2026.05.06 08:48:36.633774 [ 31856 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.633788 [ 31858 ] {} <Trace> test_15.t (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.633951 [ 31912 ] {} <Test> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Adding part 20191002_0_0_0_4 to current and virtual parts
2026.05.06 08:48:36.634045 [ 31912 ] {} <Test> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Adding part 20191002_1_1_0_4 to current and virtual parts
2026.05.06 08:48:36.634053 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 68
2026.05.06 08:48:36.634110 [ 31912 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.634192 [ 31912 ] {} <Debug> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/01414_mutations_and_errors_zookeeper_test_1/mutation_table/replicas/1/queue
2026.05.06 08:48:36.634376 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.634840 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 3, bytes: 755
2026.05.06 08:48:36.636017 [ 32244 ] {} <Test> KeeperLogStore: Flushing 3 logs
2026.05.06 08:48:36.637166 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 324
2026.05.06 08:48:36.637302 [ 31912 ] {} <Debug> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.637699 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.638404 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 2, bytes: 401
2026.05.06 08:48:36.638774 [ 31923 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.638811 [ 31911 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.639090 [ 32244 ] {} <Test> KeeperLogStore: Flushing 2 logs
2026.05.06 08:48:36.639323 [ 31910 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Creating shared ID for table test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd)
2026.05.06 08:48:36.640075 [ 31912 ] {} <Debug> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Updating strategy picker state
2026.05.06 08:48:36.640448 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 6, bytes: 986
2026.05.06 08:48:36.640754 [ 31921 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.640996 [ 31911 ] {} <Test> test_13.alter_default (ReplicatedMergeTreeQueue): Adding part all_0_0_0_3 to current and virtual parts
2026.05.06 08:48:36.641043 [ 31911 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.641092 [ 31911 ] {} <Debug> test_13.alter_default (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/test_13/test_01079/alter_default/replicas/1/queue
2026.05.06 08:48:36.641096 [ 31923 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Adding part all_2_2_0 to current and virtual parts
2026.05.06 08:48:36.641194 [ 31923 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Adding part all_1_1_0 to current and virtual parts
2026.05.06 08:48:36.641264 [ 31923 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Adding part all_0_0_0 to current and virtual parts
2026.05.06 08:48:36.641330 [ 31923 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.641446 [ 31923 ] {} <Debug> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/test_1/insert_dedup_token/replicas/r2/queue
2026.05.06 08:48:36.641659 [ 32244 ] {} <Test> KeeperLogStore: Flushing 6 logs
2026.05.06 08:48:36.642347 [ 31910 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Initializing table shared ID with 3788d1f2-ca8f-4730-aff2-1a0f511b50bd
2026.05.06 08:48:36.642421 [ 31927 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.642952 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 3, bytes: 339
2026.05.06 08:48:36.643134 [ 31864 ] {} <Information> test_15.partslost_1 (ReplicatedMergeTreeQueue): Loading 2 mutation entries: 0000000000 - 0000000001
2026.05.06 08:48:36.643301 [ 31921 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.643404 [ 32244 ] {} <Test> KeeperLogStore: Flushing 3 logs
2026.05.06 08:48:36.643394 [ 31921 ] {} <Debug> shard_1.from_1 (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/from_1_test_uasu1ifx/replicas/1/queue
2026.05.06 08:48:36.643518 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 5, bytes: 598
2026.05.06 08:48:36.643590 [ 31884 ] {} <Information> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Loading 1 mutation entries: 0000000000 - 0000000000
2026.05.06 08:48:36.643642 [ 31911 ] {} <Debug> test_13.alter_default (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.643810 [ 31923 ] {} <Debug> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.644295 [ 32244 ] {} <Test> KeeperLogStore: Flushing 5 logs
2026.05.06 08:48:36.644415 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 113
2026.05.06 08:48:36.644767 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.644857 [ 31927 ] {} <Test> test_13.rmt1 (ReplicatedMergeTreeQueue): Adding part all_0_0_0 to current and virtual parts
2026.05.06 08:48:36.644963 [ 31927 ] {} <Test> test_13.rmt1 (ReplicatedMergeTreeQueue): Adding part all_1_1_0 to current and virtual parts
2026.05.06 08:48:36.645028 [ 31927 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.645117 [ 31927 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Loading queue from /test/02485/test_13/rmt/replicas/1/queue
2026.05.06 08:48:36.645324 [ 31906 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.645451 [ 31912 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.645604 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 126
2026.05.06 08:48:36.645823 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.645878 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 125
2026.05.06 08:48:36.646127 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.646184 [ 31921 ] {} <Debug> shard_1.from_1 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.646414 [ 31911 ] {} <Debug> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Updating strategy picker state
2026.05.06 08:48:36.646529 [ 31923 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Updating strategy picker state
2026.05.06 08:48:36.646620 [ 31927 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Having 2 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.646813 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 5_2_2_0 to current and virtual parts
2026.05.06 08:48:36.646852 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 3_3_3_0 to current and virtual parts
2026.05.06 08:48:36.646880 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 2_0_0_0 to current and virtual parts
2026.05.06 08:48:36.646908 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 3_2_2_0 to current and virtual parts
2026.05.06 08:48:36.646944 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 1_0_0_0 to current and virtual parts
2026.05.06 08:48:36.646977 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 8_0_0_0 to current and virtual parts
2026.05.06 08:48:36.647005 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 4_0_0_0 to current and virtual parts
2026.05.06 08:48:36.647033 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 6_0_0_0 to current and virtual parts
2026.05.06 08:48:36.647031 [ 31884 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Adding mutation 0000000000 for 3 partitions (data versions: 20191001 = 1; 20191002 = 1; 20191003 = 1; )
2026.05.06 08:48:36.647063 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 7_0_0_0 to current and virtual parts
2026.05.06 08:48:36.647136 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 9_0_0_0 to current and virtual parts
2026.05.06 08:48:36.647174 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 0_0_0_0 to current and virtual parts
2026.05.06 08:48:36.647203 [ 31906 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.647259 [ 31906 ] {} <Debug> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Loading queue from /test/01417_freeze_partition_verbose_zookeeper_test_13/table_for_freeze_replicated/replicas/1/queue
2026.05.06 08:48:36.647445 [ 31864 ] {} <Trace> test_15.partslost_1 (ReplicatedMergeTreeQueue): Adding mutation 0000000000 for 1 partitions (data versions: all = 1; )
2026.05.06 08:48:36.647575 [ 31921 ] {} <Debug> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Updating strategy picker state
2026.05.06 08:48:36.647721 [ 31864 ] {} <Trace> test_15.partslost_1 (ReplicatedMergeTreeQueue): Adding mutation 0000000001 for 1 partitions (data versions: all = 2; )
2026.05.06 08:48:36.647969 [ 31910 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Creating shared ID for table test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd)
2026.05.06 08:48:36.648138 [ 31910 ] {} <Information> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Shared ID already set to 3788d1f2-ca8f-4730-aff2-1a0f511b50bd
2026.05.06 08:48:36.648161 [ 31927 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeQueue): Insert entry queue-0000000002 to queue with type MUTATE_PART with virtual parts [all_0_0_0_2]
2026.05.06 08:48:36.648389 [ 31906 ] {} <Debug> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.648702 [ 31911 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.648782 [ 31927 ] {} <Test> test_13.rmt1 (ReplicatedMergeTreeQueue): Adding part all_0_0_0_2 to mutations
2026.05.06 08:48:36.649151 [ 31927 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeQueue): Insert entry queue-0000000003 to queue with type MUTATE_PART with virtual parts [all_1_1_0_2]
2026.05.06 08:48:36.649249 [ 31927 ] {} <Test> test_13.rmt1 (ReplicatedMergeTreeQueue): Adding part all_1_1_0_2 to mutations
2026.05.06 08:48:36.649388 [ 31906 ] {} <Debug> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Updating strategy picker state
2026.05.06 08:48:36.649654 [ 31923 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.650824 [ 31927 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Updating strategy picker state
2026.05.06 08:48:36.651046 [ 31884 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Removing part 20191001_0_0_0_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.651051 [ 31867 ] {} <Test> test_1.test_without_merge (ReplicatedMergeTreeQueue): Removing part all_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.651051 [ 31869 ] {} <Test> test_15.src (ReplicatedMergeTreeQueue): Removing part 0_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.651183 [ 31884 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Removing part 20191002_0_0_0_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.651227 [ 31884 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Removing part 20191003_0_0_0_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.651247 [ 31869 ] {} <Test> test_15.src (ReplicatedMergeTreeQueue): Removing part 1_3_3_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.651125 [ 31910 ] {} <Information> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Became leader
2026.05.06 08:48:36.651318 [ 31869 ] {} <Test> test_15.src (ReplicatedMergeTreeQueue): Removing part 1_4_4_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.651049 [ 31864 ] {} <Test> test_15.partslost_1 (ReplicatedMergeTreeQueue): Removing part all_0_0_0_2 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.651388 [ 31869 ] {} <Test> test_15.src (ReplicatedMergeTreeQueue): Removing part 2_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.651400 [ 31910 ] {} <Trace> test_1.t (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.651351 [ 31921 ] {} <Debug> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Strategy picker state updated, current replica: 1, active replicas: [0, 1]
2026.05.06 08:48:36.651219 [ 31867 ] {} <Test> test_1.test_without_merge (ReplicatedMergeTreeQueue): Removing part all_2_2_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.651587 [ 31867 ] {} <Test> test_1.test_without_merge (ReplicatedMergeTreeQueue): Removing part all_3_3_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.651635 [ 31910 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Trying to startup table from right now
2026.05.06 08:48:36.651655 [ 31912 ] {} <Information> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Loading 1 mutation entries: 0000000002 - 0000000002
2026.05.06 08:48:36.651673 [ 31921 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.651705 [ 31910 ] {} <Debug> test_1.t (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.651782 [ 31910 ] {} <Debug> test_1.t (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.652220 [ 31906 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.653547 [ 31862 ] {} <Debug> shard_0.from_0 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.653774 [ 31944 ] {} <Trace> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.653581 [ 31858 ] {} <Debug> test_15.t (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.653856 [ 31856 ] {} <Debug> shard_1.from_0 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.653792 [ 31941 ] {} <Trace> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.653904 [ 31956 ] {} <Trace> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.653879 [ 31963 ] {} <Trace> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.653774 [ 31938 ] {} <Trace> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.653984 [ 31912 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Adding mutation 0000000002 for 1 partitions (data versions: 20191002 = 4; )
2026.05.06 08:48:36.653959 [ 31858 ] {} <Information> test_15.t (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.653943 [ 31862 ] {} <Information> shard_0.from_0 (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.653967 [ 31859 ] {} <Debug> test_15.t_r_ok (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.654073 [ 31855 ] {} <Debug> test_15.dst (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.654269 [ 31855 ] {} <Information> test_15.dst (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.654311 [ 32442 ] {} <Debug> AsyncLoader: Finish load job 'startup table shard_0.from_0' with status OK
2026.05.06 08:48:36.654340 [ 31884 ] {} <Debug> test_13.table_for_rename_replicated (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.654371 [ 31977 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Marking mutation 0000000000 done because it is <= mutation_pointer (0000000000)
2026.05.06 08:48:36.654589 [ 31856 ] {} <Information> shard_1.from_0 (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.654598 [ 31869 ] {} <Debug> test_15.src (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.654764 [ 31944 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Nothing to merge in partition 20191001: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.654806 [ 31869 ] {} <Information> test_15.src (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.654877 [ 31944 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Nothing to merge in partition 20191002: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.654909 [ 31944 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Nothing to merge in partition 20191003: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.654880 [ 31938 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92) (MergerMutator): Nothing to merge in partition 0: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.654987 [ 31864 ] {} <Debug> test_15.partslost_1 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.655027 [ 31938 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92) (MergerMutator): Nothing to merge in partition 1: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.655079 [ 31864 ] {} <Information> test_15.partslost_1 (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.655098 [ 31938 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92) (MergerMutator): Nothing to merge in partition 2: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.655272 [ 31867 ] {} <Debug> test_1.test_without_merge (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.655100 [ 31999 ] {} <Trace> test_15.partslost_1 (ReplicatedMergeTreeQueue): Marking mutation 0000000000 done because it is <= mutation_pointer (0000000001)
2026.05.06 08:48:36.655274 [ 31927 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Strategy picker state updated, current replica: 0, active replicas: [1, 2]
2026.05.06 08:48:36.655404 [ 31859 ] {} <Information> test_15.t_r_ok (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.655407 [ 31999 ] {} <Trace> test_15.partslost_1 (ReplicatedMergeTreeQueue): Marking mutation 0000000001 done because it is <= mutation_pointer (0000000001)
2026.05.06 08:48:36.655463 [ 31867 ] {} <Information> test_1.test_without_merge (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.655337 [ 31944 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Checked 3 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:36.655374 [ 31938 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92) (MergerMutator): Checked 3 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:36.655836 [ 31884 ] {} <Information> test_13.table_for_rename_replicated (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.655844 [ 32006 ] {} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55) (MergerMutator): Selected 3 parts from all_0_0_0 to all_3_3_0
2026.05.06 08:48:36.655923 [ 31927 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.655834 [ 32442 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.json_columns' in BackgrndStartup
2026.05.06 08:48:36.656071 [ 32006 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55) (MergerMutator): Checked 1 partitions, found 1 partitions with parts that may be merged: [all] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:36.656128 [ 31869 ] {} <Test> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.656272 [ 31869 ] {} <Trace> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:36.656608 [ 32442 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.json_columns' with status OK
2026.05.06 08:48:36.656730 [ 31938 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.656831 [ 32442 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.bug_67742' in BackgrndStartup
2026.05.06 08:48:36.656992 [ 32436 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.src' with status OK
2026.05.06 08:48:36.657069 [ 31911 ] {} <Information> test_13.alter_default (ReplicatedMergeTreeQueue): Loading 3 mutation entries: 0000000000 - 0000000002
2026.05.06 08:48:36.657145 [ 32436 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.parallel_replicas_plain' in BackgrndStartup
2026.05.06 08:48:36.657404 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 63
2026.05.06 08:48:36.657418 [ 32462 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.partslost_1' with status OK
2026.05.06 08:48:36.657623 [ 32462 ] {} <Debug> AsyncLoader: Execute load job 'startup table shard_1.data_02346' in BackgrndStartup
2026.05.06 08:48:36.657780 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.657813 [ 32459 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.t' with status OK
2026.05.06 08:48:36.657991 [ 32459 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.dest_view_tiny_log' in BackgrndStartup
2026.05.06 08:48:36.658131 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.dst' with status OK
2026.05.06 08:48:36.658236 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_3.unit' in BackgrndStartup
2026.05.06 08:48:36.658329 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.t_r_ok' with status OK
2026.05.06 08:48:36.658394 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.agg_func_col' in BackgrndStartup
2026.05.06 08:48:36.658519 [ 32344 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.table_for_rename_replicated' with status OK
2026.05.06 08:48:36.658768 [ 32344 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.insert_dedup_token1' in BackgrndStartup
2026.05.06 08:48:36.658869 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup table shard_1.from_0' with status OK
2026.05.06 08:48:36.658868 [ 32344 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Starting up table
2026.05.06 08:48:36.658928 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.t_modify_to_nullable' in BackgrndStartup
2026.05.06 08:48:36.659000 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.test_without_merge' with status OK
2026.05.06 08:48:36.659062 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.dynamic_test_1' in BackgrndStartup
2026.05.06 08:48:36.659118 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.dynamic_test_1' with status OK
2026.05.06 08:48:36.659154 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.data_01509' in BackgrndStartup
2026.05.06 08:48:36.659276 [ 32436 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.parallel_replicas_plain' with status OK
2026.05.06 08:48:36.659337 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 308
2026.05.06 08:48:36.659447 [ 32436 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.dst' in BackgrndStartup
2026.05.06 08:48:36.659489 [ 31912 ] {} <Test> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Removing part 20191002_0_0_0_4 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.659590 [ 31912 ] {} <Test> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Removing part 20191002_1_1_0_4 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.659702 [ 32459 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.dest_view_tiny_log' with status OK
2026.05.06 08:48:36.659829 [ 32459 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.small' in BackgrndStartup
2026.05.06 08:48:36.659919 [ 32462 ] {} <Debug> AsyncLoader: Finish load job 'startup table shard_1.data_02346' with status OK
2026.05.06 08:48:36.659974 [ 32462 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.fact_cpc_clicks' in BackgrndStartup
2026.05.06 08:48:36.660036 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_3.unit' with status OK
2026.05.06 08:48:36.660098 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.mt_00763_1' in BackgrndStartup
2026.05.06 08:48:36.660162 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.agg_func_col' with status OK
2026.05.06 08:48:36.660220 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_3.distinct_in_order' in BackgrndStartup
2026.05.06 08:48:36.660314 [ 32459 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.small' with status OK
2026.05.06 08:48:36.660387 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.660578 [ 32459 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.rmt2' in BackgrndStartup
2026.05.06 08:48:36.660630 [ 31991 ] {} <Trace> test_15.src (ReplicatedMergeTreeCleanupThread): Checking 4 blocks (4 are not cached) to clear old ones from ZooKeeper., path is /clickhouse/01034_move_partition_from_table_zookeeper_test_15/src1/blocks
2026.05.06 08:48:36.660664 [ 32459 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Starting up table
2026.05.06 08:48:36.660677 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.t_modify_to_nullable' with status OK
2026.05.06 08:48:36.660596 [ 31963 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeCleanupThread): Checking 3 blocks (3 are not cached) to clear old ones from ZooKeeper., path is /clickhouse/tables/01213_alter_rename_column_zookeeper_long_test_13/table_for_rename_replicated/blocks
2026.05.06 08:48:36.660682 [ 32029 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading 18 outdated data parts asynchronously
2026.05.06 08:48:36.660718 [ 31995 ] {} <Trace> test_15.partslost_1 (ReplicatedMergeTreeCleanupThread): Checking 1 blocks (1 are not cached) to clear old ones from ZooKeeper., path is /clickhouse/table/test_15_02067_lost/partslost/blocks
2026.05.06 08:48:36.660739 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup table 01053_db.table_for_dict' in BackgrndStartup
2026.05.06 08:48:36.660861 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.data_01509' with status OK
2026.05.06 08:48:36.661058 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.seq' in BackgrndStartup
2026.05.06 08:48:36.661067 [ 32432 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_47 from disk default
2026.05.06 08:48:36.661070 [ 32032 ] {} <Debug> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Loading 4 outdated data parts asynchronously
2026.05.06 08:48:36.661210 [ 32374 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_49 from disk default
2026.05.06 08:48:36.661218 [ 32444 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_50 from disk default
2026.05.06 08:48:36.661200 [ 32376 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_48 from disk default
2026.05.06 08:48:36.661386 [ 32398 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_52 from disk default
2026.05.06 08:48:36.661409 [ 32388 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_53 from disk default
2026.05.06 08:48:36.661529 [ 32429 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_54 from disk default
2026.05.06 08:48:36.661578 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.seq' with status OK
2026.05.06 08:48:36.661617 [ 32337 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Loading Outdated part all_4_4_0 from disk default
2026.05.06 08:48:36.661710 [ 32343 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Loading Outdated part all_3_3_0 from disk default
2026.05.06 08:48:36.661777 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test.basic' in BackgrndStartup
2026.05.06 08:48:36.661876 [ 32403 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Loading Outdated part all_1_1_0 from disk default
2026.05.06 08:48:36.661915 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test.basic' with status OK
2026.05.06 08:48:36.661957 [ 32370 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_55 from disk default
2026.05.06 08:48:36.661759 [ 31911 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Adding mutation 0000000000 for 1 partitions (data versions: all = 1; )
2026.05.06 08:48:36.661996 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.tab_00625' in BackgrndStartup
2026.05.06 08:48:36.662185 [ 32425 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_56 from disk default
2026.05.06 08:48:36.661294 [ 32381 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_51 from disk default
2026.05.06 08:48:36.662217 [ 32395 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_57 from disk default
2026.05.06 08:48:36.662393 [ 32436 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.dst' with status OK
2026.05.06 08:48:36.662514 [ 32390 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_58 from disk default
2026.05.06 08:48:36.662580 [ 32436 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.03811_sparse_column_aggregation_with_sum' in BackgrndStartup
2026.05.06 08:48:36.662734 [ 31911 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Adding mutation 0000000001 for 1 partitions (data versions: all = 2; )
2026.05.06 08:48:36.662858 [ 31911 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Adding mutation 0000000002 for 1 partitions (data versions: all = 3; )
2026.05.06 08:48:36.661780 [ 32340 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Loading Outdated part all_2_2_0 from disk default
2026.05.06 08:48:36.662962 [ 32378 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_59 from disk default
2026.05.06 08:48:36.663176 [ 32450 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_60 from disk default
2026.05.06 08:48:36.663252 [ 32436 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.03811_sparse_column_aggregation_with_sum' with status OK
2026.05.06 08:48:36.663584 [ 32436 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.src' in BackgrndStartup
2026.05.06 08:48:36.663584 [ 32035 ] {} <Debug> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2) (MergerMutator): Selected 4 parts from all_1_40_3 to all_48_48_0
2026.05.06 08:48:36.663775 [ 32436 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.src' with status OK
2026.05.06 08:48:36.663922 [ 32436 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.source_table_log' in BackgrndStartup
2026.05.06 08:48:36.663936 [ 32035 ] {} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:36.664051 [ 32436 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.source_table_log' with status OK
2026.05.06 08:48:36.664123 [ 32035 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:36.664845 [ 32402 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_61 from disk default
2026.05.06 08:48:36.664976 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Debug> MergeTask::PrepareStage: Merging 4 parts: from all_1_40_3 to all_48_48_0 into Compact with storage Full
2026.05.06 08:48:36.665291 [ 32432 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_47 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.665368 [ 32432 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_47 on disk default
2026.05.06 08:48:36.665432 [ 32368 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:36.665498 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:36.665528 [ 32429 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_54 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.664164 [ 32436 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.t_missed_subcolumns' in BackgrndStartup
2026.05.06 08:48:36.665579 [ 32432 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_47
2026.05.06 08:48:36.665629 [ 32429 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_54 on disk default
2026.05.06 08:48:36.665684 [ 32368 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:36.665733 [ 32374 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_49 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.665765 [ 32429 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_54
2026.05.06 08:48:36.665787 [ 32374 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_49 on disk default
2026.05.06 08:48:36.665833 [ 32436 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.t_missed_subcolumns' with status OK
2026.05.06 08:48:36.665858 [ 32376 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_48 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.665872 [ 32374 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_49
2026.05.06 08:48:36.665890 [ 32436 ] {} <Debug> AsyncLoader: Execute load job 'startup table shard_0.to' in BackgrndStartup
2026.05.06 08:48:36.665904 [ 32376 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_48 on disk default
2026.05.06 08:48:36.665932 [ 32436 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Starting up table
2026.05.06 08:48:36.665958 [ 32388 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_53 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.665982 [ 32376 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_48
2026.05.06 08:48:36.665998 [ 32388 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_53 on disk default
2026.05.06 08:48:36.666006 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.mt_00763_1' with status OK
2026.05.06 08:48:36.666055 [ 32403 ] {} <Test> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): loadDataPart: inserting all_1_1_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.666086 [ 32444 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_50 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.666131 [ 32403 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Finished loading Outdated part all_1_1_0 on disk default
2026.05.06 08:48:36.666130 [ 32368 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.00812003 sec. (123.152 rows/sec., 2.89 KiB/sec.)
2026.05.06 08:48:36.666168 [ 32388 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_53
2026.05.06 08:48:36.666235 [ 32368 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:36.666259 [ 32403 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758), part: all_1_1_0
2026.05.06 08:48:36.666358 [ 32368 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> HashTablesStatistics: Statistics updated for key=8003890359911665924: new sum_of_sizes=1, median_size=1
2026.05.06 08:48:36.666372 [ 32395 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_57 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.666445 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.null' in BackgrndStartup
2026.05.06 08:48:36.666140 [ 32444 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_50 on disk default
2026.05.06 08:48:36.666522 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.null' with status OK
2026.05.06 08:48:36.666539 [ 32395 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_57 on disk default
2026.05.06 08:48:36.666583 [ 32444 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_50
2026.05.06 08:48:36.666603 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.input_function_table_3' in BackgrndStartup
2026.05.06 08:48:36.666711 [ 32007 ] {} <Trace> test_1.test_without_merge (ReplicatedMergeTreeCleanupThread): Checking 3 blocks (3 are not cached) to clear old ones from ZooKeeper., path is /clickhouse/tables/test_1/test02676_without_merge/blocks
2026.05.06 08:48:36.666782 [ 32462 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.fact_cpc_clicks' with status OK
2026.05.06 08:48:36.666777 [ 32395 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_57
2026.05.06 08:48:36.666838 [ 32343 ] {} <Test> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): loadDataPart: inserting all_3_3_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.666852 [ 32462 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.simple_key_simple_attributes_source_table' in BackgrndStartup
2026.05.06 08:48:36.666891 [ 32343 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Finished loading Outdated part all_3_3_0 on disk default
2026.05.06 08:48:36.666934 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.tab_00625' with status OK
2026.05.06 08:48:36.666972 [ 32343 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758), part: all_3_3_0
2026.05.06 08:48:36.666989 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.tab' in BackgrndStartup
2026.05.06 08:48:36.667099 [ 32378 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_59 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.667098 [ 32442 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.bug_67742' with status OK
2026.05.06 08:48:36.667156 [ 32378 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_59 on disk default
2026.05.06 08:48:36.667199 [ 32450 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_60 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.667237 [ 32378 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_59
2026.05.06 08:48:36.667232 [ 32442 ] {} <Debug> AsyncLoader: Execute load job 'startup table shard_0.from_1' in BackgrndStartup
2026.05.06 08:48:36.667262 [ 32450 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_60 on disk default
2026.05.06 08:48:36.667376 [ 32390 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_58 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.667428 [ 32390 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_58 on disk default
2026.05.06 08:48:36.667428 [ 32442 ] {} <Trace> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Starting up table
2026.05.06 08:48:36.667510 [ 32337 ] {} <Test> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): loadDataPart: inserting all_4_4_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.667518 [ 32390 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_58
2026.05.06 08:48:36.667554 [ 32450 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_60
2026.05.06 08:48:36.667586 [ 32337 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Finished loading Outdated part all_4_4_0 on disk default
2026.05.06 08:48:36.667564 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_3.distinct_in_order' with status OK
2026.05.06 08:48:36.667656 [ 32340 ] {} <Test> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): loadDataPart: inserting all_2_2_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.667688 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.temp' in BackgrndStartup
2026.05.06 08:48:36.667694 [ 32337 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758), part: all_4_4_0
2026.05.06 08:48:36.667709 [ 32340 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Finished loading Outdated part all_2_2_0 on disk default
2026.05.06 08:48:36.667756 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup table 01053_db.table_for_dict' with status OK
2026.05.06 08:48:36.667799 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758), part: all_2_2_0
2026.05.06 08:48:36.667821 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_3.parsing_with_names' in BackgrndStartup
2026.05.06 08:48:36.667902 [ 32032 ] {} <Debug> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Loaded 4 outdated data parts asynchronously
2026.05.06 08:48:36.667924 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_3.parsing_with_names' with status OK
2026.05.06 08:48:36.667930 [ 32340 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_62 from disk default
2026.05.06 08:48:36.668011 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.dict_02916_dictionary_access_test_13' in BackgrndStartup
2026.05.06 08:48:36.667708 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_40_3, total 40000 rows starting from the beginning of the part
2026.05.06 08:48:36.668045 [ 32451 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_64 from disk default
2026.05.06 08:48:36.668060 [ 32381 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_51 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.668071 [ 32462 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.simple_key_simple_attributes_source_table' with status OK
2026.05.06 08:48:36.668069 [ 32460 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loading Outdated part all_1_1_0_63 from disk default
2026.05.06 08:48:36.668117 [ 32381 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_51 on disk default
2026.05.06 08:48:36.668150 [ 32462 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.storage' in BackgrndStartup
2026.05.06 08:48:36.668140 [ 32370 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_55 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.668200 [ 32381 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_51
2026.05.06 08:48:36.668277 [ 32462 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.storage' with status OK
2026.05.06 08:48:36.668285 [ 32370 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_55 on disk default
2026.05.06 08:48:36.668297 [ 32425 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_56 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.668318 [ 32462 ] {} <Debug> AsyncLoader: Execute load job 'startup table test.hits_buffer' in BackgrndStartup
2026.05.06 08:48:36.668407 [ 32425 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_56 on disk default
2026.05.06 08:48:36.668456 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.temp' with status OK
2026.05.06 08:48:36.668460 [ 32370 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_55
2026.05.06 08:48:36.668519 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.simple_key_source_table_01862' in BackgrndStartup
2026.05.06 08:48:36.668565 [ 32425 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_56
2026.05.06 08:48:36.668591 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.input_function_table_3' with status OK
2026.05.06 08:48:36.668649 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.test_grouping_sets_predicate' in BackgrndStartup
2026.05.06 08:48:36.668709 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.dict_02916_dictionary_access_test_13' with status OK
2026.05.06 08:48:36.668754 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.mt_with_pk' in BackgrndStartup
2026.05.06 08:48:36.668782 [ 32398 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_52 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.668772 [ 32391 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.668844 [ 32398 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_52 on disk default
2026.05.06 08:48:36.668827 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_41_46_1, total 6000 rows starting from the beginning of the part
2026.05.06 08:48:36.668895 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.test_grouping_sets_predicate' with status OK
2026.05.06 08:48:36.668922 [ 32398 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_52
2026.05.06 08:48:36.668943 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_02771.t' in BackgrndStartup
2026.05.06 08:48:36.669109 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.mt_with_pk' with status OK
2026.05.06 08:48:36.669159 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.02183_dictionary_test_table' in BackgrndStartup
2026.05.06 08:48:36.669211 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.02183_dictionary_test_table' with status OK
2026.05.06 08:48:36.669214 [ 32040 ] {} <Debug> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Loading 5 outdated data parts asynchronously
2026.05.06 08:48:36.669246 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.test' in BackgrndStartup
2026.05.06 08:48:36.669317 [ 32362 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.669378 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_47_47_0, total 1000 rows starting from the beginning of the part
2026.05.06 08:48:36.669372 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.simple_key_source_table_01862' with status OK
2026.05.06 08:48:36.669421 [ 32476 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Loading Outdated part 201810_4_4_0 from disk default
2026.05.06 08:48:36.669569 [ 32454 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Loading Outdated part 201810_1_1_0 from disk default
2026.05.06 08:48:36.669611 [ 32456 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Loading Outdated part 201810_2_2_0 from disk default
2026.05.06 08:48:36.669664 [ 32463 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Loading Outdated part 201810_1_2_1 from disk default
2026.05.06 08:48:36.669678 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.source_table_stripe_log' in BackgrndStartup
2026.05.06 08:48:36.669747 [ 32466 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Loading Outdated part 201810_1_2_1_3 from disk default
2026.05.06 08:48:36.669810 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_02771.t' with status OK
2026.05.06 08:48:36.669866 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.dest_view_log' in BackgrndStartup
2026.05.06 08:48:36.669918 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.dest_view_log' with status OK
2026.05.06 08:48:36.670036 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.A_M' in BackgrndStartup
2026.05.06 08:48:36.670076 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.A_M' with status OK
2026.05.06 08:48:36.670066 [ 32384 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.670122 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.02162_test_table' in BackgrndStartup
2026.05.06 08:48:36.670141 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_48_48_0, total 1000 rows starting from the beginning of the part
2026.05.06 08:48:36.670172 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.02162_test_table' with status OK
2026.05.06 08:48:36.670208 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table 01778_db.hierarchy_flat_dictionary' in BackgrndStartup
2026.05.06 08:48:36.670264 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table 01778_db.hierarchy_flat_dictionary' with status OK
2026.05.06 08:48:36.670305 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.override_test__fuzz_45' in BackgrndStartup
2026.05.06 08:48:36.670549 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.override_test__fuzz_45' with status OK
2026.05.06 08:48:36.670617 [ 32402 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_61 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.670627 [ 32461 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.670634 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_truncate_database.dest_dictionary' in BackgrndStartup
2026.05.06 08:48:36.670734 [ 32402 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_61 on disk default
2026.05.06 08:48:36.670785 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.dest_dictionary' with status OK
2026.05.06 08:48:36.670835 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.bug_13492' in BackgrndStartup
2026.05.06 08:48:36.670878 [ 32402 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_61
2026.05.06 08:48:36.670884 [ 32451 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_64 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.670967 [ 32451 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_64 on disk default
2026.05.06 08:48:36.671046 [ 32451 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_64
2026.05.06 08:48:36.671168 [ 32340 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_62 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.671266 [ 32340 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_62 on disk default
2026.05.06 08:48:36.671405 [ 32340 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_62
2026.05.06 08:48:36.671453 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.bug_13492' with status OK
2026.05.06 08:48:36.671509 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.dict_01509' in BackgrndStartup
2026.05.06 08:48:36.671565 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.dict_01509' with status OK
2026.05.06 08:48:36.671614 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table 01053_db.ssd_dict' in BackgrndStartup
2026.05.06 08:48:36.671695 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table 01053_db.ssd_dict' with status OK
2026.05.06 08:48:36.671769 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.simple_key_hashed_dictionary_01862' in BackgrndStartup
2026.05.06 08:48:36.671816 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.simple_key_hashed_dictionary_01862' with status OK
2026.05.06 08:48:36.671855 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.hashed_array_dictionary_simple_key_simple_attributes' in BackgrndStartup
2026.05.06 08:48:36.671906 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.hashed_array_dictionary_simple_key_simple_attributes' with status OK
2026.05.06 08:48:36.671939 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_15.02183_hashed_array_dictionary' in BackgrndStartup
2026.05.06 08:48:36.671997 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_15.02183_hashed_array_dictionary' with status OK
2026.05.06 08:48:36.672033 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.tab2' in BackgrndStartup
2026.05.06 08:48:36.672221 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.tab2' with status OK
2026.05.06 08:48:36.672260 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup table shard_1.to' in BackgrndStartup
2026.05.06 08:48:36.672334 [ 32439 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Starting up table
2026.05.06 08:48:36.672424 [ 32460 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): loadDataPart: inserting all_1_1_0_63 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.672427 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.test' with status OK
2026.05.06 08:48:36.672495 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup table 01945.db.test_dictionary' in BackgrndStartup
2026.05.06 08:48:36.672495 [ 32460 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Finished loading Outdated part all_1_1_0_63 on disk default
2026.05.06 08:48:36.672562 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.tab' with status OK
2026.05.06 08:48:36.672586 [ 32460 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0), part: all_1_1_0_63
2026.05.06 08:48:36.672627 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1..inner_id.9907701c-369a-410f-99dc-ca9afd271de1' in BackgrndStartup
2026.05.06 08:48:36.672698 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_truncate_database.source_table_stripe_log' with status OK
2026.05.06 08:48:36.672829 [ 32029 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Loaded 18 outdated data parts asynchronously
2026.05.06 08:48:36.672875 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_13.02162_test_dictionary' in BackgrndStartup
2026.05.06 08:48:36.672953 [ 32462 ] {} <Debug> AsyncLoader: Finish load job 'startup table test.hits_buffer' with status OK
2026.05.06 08:48:36.673082 [ 32462 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.test2' in BackgrndStartup
2026.05.06 08:48:36.673144 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup table 01945.db.test_dictionary' with status OK
2026.05.06 08:48:36.673199 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.concurrent_alter_column' in BackgrndStartup
2026.05.06 08:48:36.673263 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1..inner_id.9907701c-369a-410f-99dc-ca9afd271de1' with status OK
2026.05.06 08:48:36.673321 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test.hits_dst' in BackgrndStartup
2026.05.06 08:48:36.673377 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.02162_test_dictionary' with status OK
2026.05.06 08:48:36.673431 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup table test.hits_s3' in BackgrndStartup
2026.05.06 08:48:36.673496 [ 32462 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.test2' with status OK
2026.05.06 08:48:36.673546 [ 32462 ] {} <Debug> AsyncLoader: Execute load job 'startup table test.hits' in BackgrndStartup
2026.05.06 08:48:36.673628 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test.hits_dst' with status OK
2026.05.06 08:48:36.673748 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup table test_1.test_s3_mt' in BackgrndStartup
2026.05.06 08:48:36.673805 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup table test.hits_s3' with status OK
2026.05.06 08:48:36.673857 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database default' in BackgrndStartup
2026.05.06 08:48:36.673911 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database default' with status OK
2026.05.06 08:48:36.673947 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database rdb_test_13_3' in BackgrndStartup
2026.05.06 08:48:36.673982 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database rdb_test_13_3' with status OK
2026.05.06 08:48:36.674022 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database rdb_test_13_9' in BackgrndStartup
2026.05.06 08:48:36.674055 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database rdb_test_13_9' with status OK
2026.05.06 08:48:36.674046 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:36.674090 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database datasets' in BackgrndStartup
2026.05.06 08:48:36.674190 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database datasets' with status OK
2026.05.06 08:48:36.674227 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_11' in BackgrndStartup
2026.05.06 08:48:36.674260 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_11' with status OK
2026.05.06 08:48:36.674302 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database rdb_test_13_2' in BackgrndStartup
2026.05.06 08:48:36.674335 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database rdb_test_13_2' with status OK
2026.05.06 08:48:36.674368 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_5' in BackgrndStartup
2026.05.06 08:48:36.674400 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_5' with status OK
2026.05.06 08:48:36.674433 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_mkf6rch8' in BackgrndStartup
2026.05.06 08:48:36.674465 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_mkf6rch8' with status OK
2026.05.06 08:48:36.674539 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_01516' in BackgrndStartup
2026.05.06 08:48:36.674602 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_01516' with status OK
2026.05.06 08:48:36.674635 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_2gxskyku_1' in BackgrndStartup
2026.05.06 08:48:36.674674 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_2gxskyku_1' with status OK
2026.05.06 08:48:36.674709 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_w50hoezt' in BackgrndStartup
2026.05.06 08:48:36.674741 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_w50hoezt' with status OK
2026.05.06 08:48:36.674760 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:36.674775 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_m91yt8r2_1' in BackgrndStartup
2026.05.06 08:48:36.674840 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_m91yt8r2_1' with status OK
2026.05.06 08:48:36.674875 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_jbjeaal4_1' in BackgrndStartup
2026.05.06 08:48:36.674942 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_jbjeaal4_1' with status OK
2026.05.06 08:48:36.675005 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database rdb_test_13' in BackgrndStartup
2026.05.06 08:48:36.675038 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database rdb_test_13' with status OK
2026.05.06 08:48:36.675073 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_xfltqef0' in BackgrndStartup
2026.05.06 08:48:36.675105 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_xfltqef0' with status OK
2026.05.06 08:48:36.675137 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_3' in BackgrndStartup
2026.05.06 08:48:36.675173 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_3' with status OK
2026.05.06 08:48:36.675185 [ 32476 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): loadDataPart: inserting 201810_4_4_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.675205 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_02771' in BackgrndStartup
2026.05.06 08:48:36.675261 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_02771' with status OK
2026.05.06 08:48:36.675294 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database 01778_db' in BackgrndStartup
2026.05.06 08:48:36.675292 [ 32476 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Finished loading Outdated part 201810_4_4_0 on disk default
2026.05.06 08:48:36.675322 [ 32454 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): loadDataPart: inserting 201810_1_1_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.675363 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database 01778_db' with status OK
2026.05.06 08:48:36.675437 [ 32454 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Finished loading Outdated part 201810_1_1_0 on disk default
2026.05.06 08:48:36.675458 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database 01053_db' in BackgrndStartup
2026.05.06 08:48:36.675507 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database 01053_db' with status OK
2026.05.06 08:48:36.675543 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_15' in BackgrndStartup
2026.05.06 08:48:36.675547 [ 32476 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2), part: 201810_4_4_0
2026.05.06 08:48:36.675577 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_15' with status OK
2026.05.06 08:48:36.675632 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_truncate_database' in BackgrndStartup
2026.05.06 08:48:36.675623 [ 32454 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2), part: 201810_1_1_0
2026.05.06 08:48:36.675677 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_truncate_database' with status OK
2026.05.06 08:48:36.675714 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database 01945.db' in BackgrndStartup
2026.05.06 08:48:36.675822 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database 01945.db' with status OK
2026.05.06 08:48:36.675856 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Replicated database rdb_test_13_3' in BackgrndStartup
2026.05.06 08:48:36.675908 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.concurrent_alter_column' with status OK
2026.05.06 08:48:36.675935 [ 32438 ] {} <Debug> DatabaseReplicated (rdb_test_13_3): Calculated metadata digest of 0 tables: 0
2026.05.06 08:48:36.675967 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Replicated database rdb_test_13_9' in BackgrndStartup
2026.05.06 08:48:36.676042 [ 32424 ] {} <Debug> DatabaseReplicated (rdb_test_13_9): Calculated metadata digest of 0 tables: 0
2026.05.06 08:48:36.676055 [ 32462 ] {} <Debug> AsyncLoader: Finish load job 'startup table test.hits' with status OK
2026.05.06 08:48:36.676122 [ 32462 ] {} <Debug> AsyncLoader: Execute load job 'startup Replicated database test_11' in BackgrndStartup
2026.05.06 08:48:36.676177 [ 32462 ] {} <Debug> DatabaseReplicated (test_11): Calculated metadata digest of 0 tables: 0
2026.05.06 08:48:36.676369 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Replicated database rdb_test_13_3' with status OK
2026.05.06 08:48:36.676420 [ 32468 ] {} <Debug> DDLWorker(rdb_test_13_3): Starting DDLWorker thread
2026.05.06 08:48:36.676456 [ 32477 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:36.676495 [ 32438 ] {} <Debug> AsyncLoader: Execute load job 'startup Replicated database rdb_test_13_2' in BackgrndStartup
2026.05.06 08:48:36.676501 [ 32419 ] {} <Debug> DDLWorker(rdb_test_13_9): Starting DDLWorker thread
2026.05.06 08:48:36.676477 [ 32421 ] {} <Debug> DDLWorker(rdb_test_13_3): Started DDLWorker cleanup thread
2026.05.06 08:48:36.676544 [ 32438 ] {} <Debug> DatabaseReplicated (rdb_test_13_2): Calculated metadata digest of 0 tables: 0
2026.05.06 08:48:36.676556 [ 32441 ] {} <Debug> DDLWorker(rdb_test_13_9): Started DDLWorker cleanup thread
2026.05.06 08:48:36.676421 [ 32445 ] {} <Debug> DDLWorker(test_11): Starting DDLWorker thread
2026.05.06 08:48:36.676708 [ 32431 ] {} <Debug> DDLWorker(test_11): Started DDLWorker cleanup thread
2026.05.06 08:48:36.676698 [ 32462 ] {} <Debug> AsyncLoader: Finish load job 'startup Replicated database test_11' with status OK
2026.05.06 08:48:36.676755 [ 32470 ] {} <Debug> DDLWorker(rdb_test_13_2): Starting DDLWorker thread
2026.05.06 08:48:36.676798 [ 32471 ] {} <Debug> DDLWorker(rdb_test_13_2): Started DDLWorker cleanup thread
2026.05.06 08:48:36.676944 [ 32462 ] {} <Debug> AsyncLoader: Execute load job 'startup Replicated database test_5' in BackgrndStartup
2026.05.06 08:48:36.676939 [ 32477 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:36 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE7291D4E5518
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:36.676955 [ 32456 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): loadDataPart: inserting 201810_2_2_0 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.677027 [ 32462 ] {} <Debug> DatabaseReplicated (test_5): Calculated metadata digest of 0 tables: 0
2026.05.06 08:48:36.677036 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Replicated database rdb_test_13_9' with status OK
2026.05.06 08:48:36.677140 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_mkf6rch8' in BackgrndStartup
2026.05.06 08:48:36.677142 [ 32456 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Finished loading Outdated part 201810_2_2_0 on disk default
2026.05.06 08:48:36.677191 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_mkf6rch8' with status OK
2026.05.06 08:48:36.677241 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_2gxskyku_1' in BackgrndStartup
2026.05.06 08:48:36.677264 [ 32477 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:36.677336 [ 32440 ] {} <Debug> DDLWorker(test_5): Starting DDLWorker thread
2026.05.06 08:48:36.677347 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_2gxskyku_1' with status OK
2026.05.06 08:48:36.677354 [ 32453 ] {} <Debug> DDLWorker(test_5): Started DDLWorker cleanup thread
2026.05.06 08:48:36.677355 [ 32456 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2), part: 201810_2_2_0
2026.05.06 08:48:36.677384 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_w50hoezt' in BackgrndStartup
2026.05.06 08:48:36.677418 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_w50hoezt' with status OK
2026.05.06 08:48:36.677458 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_m91yt8r2_1' in BackgrndStartup
2026.05.06 08:48:36.677498 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_m91yt8r2_1' with status OK
2026.05.06 08:48:36.677529 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_jbjeaal4_1' in BackgrndStartup
2026.05.06 08:48:36.677593 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_jbjeaal4_1' with status OK
2026.05.06 08:48:36.677593 [ 32466 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): loadDataPart: inserting 201810_1_2_1_3 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.677649 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database rdb_test_13' in BackgrndStartup
2026.05.06 08:48:36.677684 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database rdb_test_13' with status OK
2026.05.06 08:48:36.677720 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_xfltqef0' in BackgrndStartup
2026.05.06 08:48:36.677721 [ 32463 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): loadDataPart: inserting 201810_1_2_1 (state Outdated) into data_parts_indexes
2026.05.06 08:48:36.677728 [ 32466 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Finished loading Outdated part 201810_1_2_1_3 on disk default
2026.05.06 08:48:36.677752 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_xfltqef0' with status OK
2026.05.06 08:48:36.677780 [ 32463 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Finished loading Outdated part 201810_1_2_1 on disk default
2026.05.06 08:48:36.677837 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_3' in BackgrndStartup
2026.05.06 08:48:36.677869 [ 32463 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2), part: 201810_1_2_1
2026.05.06 08:48:36.677872 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_3' with status OK
2026.05.06 08:48:36.677915 [ 32466 ] {} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2), part: 201810_1_2_1_3
2026.05.06 08:48:36.677968 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_02771' in BackgrndStartup
2026.05.06 08:48:36.678092 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_02771' with status OK
2026.05.06 08:48:36.678100 [ 32040 ] {} <Debug> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Loaded 5 outdated data parts asynchronously
2026.05.06 08:48:36.678168 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database 01778_db' in BackgrndStartup
2026.05.06 08:48:36.678205 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database 01778_db' with status OK
2026.05.06 08:48:36.678239 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database 01053_db' in BackgrndStartup
2026.05.06 08:48:36.678241 [ 31910 ] {} <Trace> test_1.t (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.678271 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database 01053_db' with status OK
2026.05.06 08:48:36.678308 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_15' in BackgrndStartup
2026.05.06 08:48:36.678340 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_15' with status OK
2026.05.06 08:48:36.678372 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_truncate_database' in BackgrndStartup
2026.05.06 08:48:36.678444 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_truncate_database' with status OK
2026.05.06 08:48:36.678519 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database 01945.db' in BackgrndStartup
2026.05.06 08:48:36.678553 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database 01945.db' with status OK
2026.05.06 08:48:36.678584 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test' in BackgrndStartup
2026.05.06 08:48:36.678616 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test' with status OK
2026.05.06 08:48:36.678649 [ 32424 ] {} <Debug> AsyncLoader: Execute load job 'startup Replicated database rdb_test_13' in BackgrndStartup
2026.05.06 08:48:36.678746 [ 32424 ] {} <Debug> DatabaseReplicated (rdb_test_13): Calculated metadata digest of 1 tables: 3663615713908833203
2026.05.06 08:48:36.678753 [ 31912 ] {} <Debug> test_1.replicated_mutation_table (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.678742 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.test_s3_mt' with status OK
2026.05.06 08:48:36.678828 [ 31912 ] {} <Information> test_1.replicated_mutation_table (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.678893 [ 32477 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test' in BackgrndStartup
2026.05.06 08:48:36.678846 [ 32062 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Marking mutation 0000000002 done because it is <= mutation_pointer (0000000002)
2026.05.06 08:48:36.679080 [ 32472 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.replicated_mutation_table' with status OK
2026.05.06 08:48:36.679118 [ 32061 ] {} <Test> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3) (MergerMutator): Nothing to merge in partition 20191002: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.679178 [ 32472 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:36.679209 [ 32061 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:36.679278 [ 32462 ] {} <Debug> AsyncLoader: Finish load job 'startup Replicated database test_5' with status OK
2026.05.06 08:48:36.679330 [ 32462 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:36.679409 [ 32424 ] {} <Debug> AsyncLoader: Finish load job 'startup Replicated database rdb_test_13' with status OK
2026.05.06 08:48:36.679464 [ 32424 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:36.679541 [ 32438 ] {} <Debug> AsyncLoader: Finish load job 'startup Replicated database rdb_test_13_2' with status OK
2026.05.06 08:48:36.679749 [ 32438 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:36.679856 [ 32477 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test' with status OK
2026.05.06 08:48:36.679883 [ 32419 ] {} <Warning> ZooKeeper: Ephemeral node ('/test/test_13/rdb/replicas/s9|r9/active') already exists but it isn't owned by us. Will wait until it disappears
2026.05.06 08:48:36.679956 [ 32477 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:36.680119 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 2, bytes: 93
2026.05.06 08:48:36.680241 [ 31923 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Removing part all_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.680257 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:36.680294 [ 31911 ] {} <Test> test_13.alter_default (ReplicatedMergeTreeQueue): Removing part all_0_0_0_3 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.680288 [ 31923 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Removing part all_1_1_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.680333 [ 31910 ] {} <Test> test_1.t (ReplicatedMergeTreeQueue): Adding part all_0_0_0 to current and virtual parts
2026.05.06 08:48:36.680448 [ 31923 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Removing part all_2_2_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.680467 [ 31910 ] {} <Test> test_1.t (ReplicatedMergeTreeQueue): Adding part all_0_0_1 to current and virtual parts
2026.05.06 08:48:36.680564 [ 31910 ] {} <Test> test_1.t (ReplicatedMergeTreeQueue): Adding part all_0_0_1_1 to current and virtual parts
2026.05.06 08:48:36.680626 [ 31910 ] {} <Trace> test_1.t (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.680683 [ 32244 ] {} <Test> KeeperLogStore: Flushing 2 logs
2026.05.06 08:48:36.680716 [ 31910 ] {} <Debug> test_1.t (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/test/test_1/test_02477/replicas/1/queue
2026.05.06 08:48:36.680806 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 2, bytes: 92
2026.05.06 08:48:36.680863 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.682646 [ 32244 ] {} <Test> KeeperLogStore: Flushing 2 logs
2026.05.06 08:48:36.683357 [ 31927 ] {} <Information> test_13.rmt1 (ReplicatedMergeTreeQueue): Loading 1 mutation entries: 0000000000 - 0000000000
2026.05.06 08:48:36.683801 [ 31910 ] {} <Debug> test_1.t (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.684153 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 89
2026.05.06 08:48:36.684356 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.684731 [ 31911 ] {} <Debug> test_13.alter_default (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.684835 [ 31911 ] {} <Information> test_13.alter_default (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.684840 [ 32068 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Marking mutation 0000000000 done because it is <= mutation_pointer (0000000002)
2026.05.06 08:48:36.684870 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 67
2026.05.06 08:48:36.684922 [ 32068 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Marking mutation 0000000001 done because it is <= mutation_pointer (0000000002)
2026.05.06 08:48:36.684950 [ 32068 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Marking mutation 0000000002 done because it is <= mutation_pointer (0000000002)
2026.05.06 08:48:36.684940 [ 31923 ] {} <Debug> test_1.insert_dedup_token2 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.685018 [ 32418 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.alter_default' with status OK
2026.05.06 08:48:36.685143 [ 31923 ] {} <Information> test_1.insert_dedup_token2 (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.685119 [ 32070 ] {} <Test> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.685152 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.685172 [ 31641 ] {} <Test> TCPHandler: Closing connection (open: true, cancelled: false, eof: true)
2026.05.06 08:48:36.685181 [ 31927 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeQueue): Adding mutation 0000000000 for 1 partitions (data versions: all = 2; )
2026.05.06 08:48:36.685231 [ 32418 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:36.685254 [ 31641 ] {} <Debug> TCPHandler: Done processing connection.
2026.05.06 08:48:36.685235 [ 32070 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:36.685330 [ 31911 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Selected 3 parts from all_0_0_0 to all_2_2_0
2026.05.06 08:48:36.685380 [ 32446 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.insert_dedup_token2' with status OK
2026.05.06 08:48:36.685439 [ 32446 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:36.685570 [ 31910 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Updating strategy picker state
2026.05.06 08:48:36.685762 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 79
2026.05.06 08:48:36.686162 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.693401 [ 31921 ] {} <Debug> shard_1.from_1 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.693475 [ 31921 ] {} <Information> shard_1.from_1 (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.693606 [ 32457 ] {} <Debug> AsyncLoader: Finish load job 'startup table shard_1.from_1' with status OK
2026.05.06 08:48:36.693672 [ 32457 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:36.685467 [ 31911 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Checked 1 partitions, found 1 partitions with parts that may be merged: [all] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:36.694579 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 83
2026.05.06 08:48:36.694838 [ 31641 ] {} <Debug> TCP-Session-50c31820-497d-459e-a5ab-c5bf6fa935e8: 50c31820-497d-459e-a5ab-c5bf6fa935e8 Logout, user_id: 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:36.694984 [ 32071 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeCleanupThread): Checking 1 blocks (1 are not cached) to clear old ones from ZooKeeper., path is /clickhouse/tables/test_13/test_01079/alter_default/blocks
2026.05.06 08:48:36.695035 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 79
2026.05.06 08:48:36.695005 [ 32090 ] {} <Trace> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.695198 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.695321 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 0_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.695372 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 1_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.695325 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.695402 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 2_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.695432 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 3_2_2_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.695463 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 3_3_3_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.695501 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 4_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.695529 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 5_2_2_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.695562 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 6_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.695588 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 7_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.695640 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 8_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.695668 [ 31906 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 9_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.696494 [ 31910 ] {} <Trace> test_1.t (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.696589 [ 32079 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeCleanupThread): Checking 3 blocks (3 are not cached) to clear old ones from ZooKeeper., path is /clickhouse/tables/test_1/insert_dedup_token/blocks
2026.05.06 08:48:36.697171 [ 31927 ] {} <Test> test_13.rmt1 (ReplicatedMergeTreeQueue): Removing part all_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.697219 [ 31927 ] {} <Test> test_13.rmt1 (ReplicatedMergeTreeQueue): Removing part all_1_1_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.697234 [ 32006 ] {} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55) (MergerMutator): Selected 3 parts from all_0_0_0 to all_3_3_0
2026.05.06 08:48:36.698783 [ 31944 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.698791 [ 31869 ] {} <Trace> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.699317 [ 32445 ] {} <Trace> DDLWorker(test_11): Trying to initialize replication: our_log_ptr=1, max_log_ptr=1, local_digest=0, zk_digest=0
2026.05.06 08:48:36.699319 [ 32440 ] {} <Trace> DDLWorker(test_5): Trying to initialize replication: our_log_ptr=1, max_log_ptr=1, local_digest=0, zk_digest=0
2026.05.06 08:48:36.699318 [ 32470 ] {} <Trace> DDLWorker(rdb_test_13_2): Trying to initialize replication: our_log_ptr=1, max_log_ptr=1, local_digest=0, zk_digest=0
2026.05.06 08:48:36.699497 [ 32445 ] {} <Test> DatabaseReplicated (test_11): Current in-memory metadata digest: 0
2026.05.06 08:48:36.699497 [ 32470 ] {} <Test> DatabaseReplicated (rdb_test_13_2): Current in-memory metadata digest: 0
2026.05.06 08:48:36.699498 [ 32440 ] {} <Test> DatabaseReplicated (test_5): Current in-memory metadata digest: 0
2026.05.06 08:48:36.700074 [ 32468 ] {} <Trace> DDLWorker(rdb_test_13_3): Trying to initialize replication: our_log_ptr=1, max_log_ptr=1, local_digest=0, zk_digest=0
2026.05.06 08:48:36.700251 [ 31906 ] {} <Debug> test_13.table_for_freeze_replicated (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.700331 [ 31906 ] {} <Information> test_13.table_for_freeze_replicated (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.700327 [ 32468 ] {} <Test> DatabaseReplicated (rdb_test_13_3): Current in-memory metadata digest: 0
2026.05.06 08:48:36.700448 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 138
2026.05.06 08:48:36.700451 [ 32095 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 0: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.700517 [ 32095 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 1: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.700508 [ 32342 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.table_for_freeze_replicated' with status OK
2026.05.06 08:48:36.700547 [ 32095 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 2: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.700577 [ 32095 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 3: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.700605 [ 32095 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 4: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.700633 [ 32095 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 5: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.700623 [ 32342 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:36.700668 [ 32095 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 6: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.700696 [ 32095 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 7: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.700723 [ 32095 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 8: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.700750 [ 32095 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 9: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.700798 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.700846 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 4, bytes: 747
2026.05.06 08:48:36.700805 [ 32095 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Checked 10 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:36.700960 [ 32095 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.701018 [ 31927 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.701134 [ 31927 ] {} <Information> test_13.rmt1 (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.701300 [ 32038 ] {} <Debug> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Creating shared ID for table shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf)
2026.05.06 08:48:36.701329 [ 32464 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.rmt1' with status OK
2026.05.06 08:48:36.701412 [ 32464 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:36.701434 [ 32104 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:36.701616 [ 32244 ] {} <Test> KeeperLogStore: Flushing 4 logs
2026.05.06 08:48:36.701986 [ 32470 ] {} <Debug> DDLWorker(rdb_test_13_2): Initialized DDLWorker thread
2026.05.06 08:48:36.702145 [ 32440 ] {} <Debug> DDLWorker(test_5): Initialized DDLWorker thread
2026.05.06 08:48:36.702143 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:36.702192 [ 32445 ] {} <Debug> DDLWorker(test_11): Initialized DDLWorker thread
2026.05.06 08:48:36.702208 [ 32104 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:36.702213 [ 32470 ] {} <Debug> DDLWorker(rdb_test_13_2): Scheduling tasks
2026.05.06 08:48:36.702249 [ 32471 ] {} <Debug> DDLWorker(rdb_test_13_2): Cleaning queue
2026.05.06 08:48:36.702277 [ 32453 ] {} <Debug> DDLWorker(test_5): Cleaning queue
2026.05.06 08:48:36.702256 [ 32445 ] {} <Debug> DDLWorker(test_11): Scheduling tasks
2026.05.06 08:48:36.702276 [ 32431 ] {} <Debug> DDLWorker(test_11): Cleaning queue
2026.05.06 08:48:36.702224 [ 32440 ] {} <Debug> DDLWorker(test_5): Scheduling tasks
2026.05.06 08:48:36.702432 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:36.702464 [ 32470 ] {} <Trace> DDLWorker(rdb_test_13_2): Don't have unfinished tasks after restarting
2026.05.06 08:48:36.702494 [ 32440 ] {} <Trace> DDLWorker(test_5): Don't have unfinished tasks after restarting
2026.05.06 08:48:36.702604 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 5, bytes: 560
2026.05.06 08:48:36.702693 [ 32445 ] {} <Trace> DDLWorker(test_11): Don't have unfinished tasks after restarting
2026.05.06 08:48:36.702737 [ 32006 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Created log entry /clickhouse/tables/test_1/test02676_without_merge/log/log-0000000003 for merge all_0_3_1
2026.05.06 08:48:36.702952 [ 32244 ] {} <Test> KeeperLogStore: Flushing 5 logs
2026.05.06 08:48:36.703116 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Creating shared ID for table test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1)
2026.05.06 08:48:36.703200 [ 32006 ] {} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55) (MergerMutator): Selected 3 parts from all_0_0_0 to all_3_3_0
2026.05.06 08:48:36.703384 [ 32006 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55) (MergerMutator): Checked 1 partitions, found 1 partitions with parts that may be merged: [all] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:36.703446 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:36.703637 [ 32468 ] {} <Debug> DDLWorker(rdb_test_13_3): Initialized DDLWorker thread
2026.05.06 08:48:36.703661 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 5, bytes: 605
2026.05.06 08:48:36.703702 [ 32468 ] {} <Debug> DDLWorker(rdb_test_13_3): Scheduling tasks
2026.05.06 08:48:36.703746 [ 32421 ] {} <Debug> DDLWorker(rdb_test_13_3): Cleaning queue
2026.05.06 08:48:36.703752 [ 32468 ] {} <Trace> DDLWorker(rdb_test_13_3): Don't have unfinished tasks after restarting
2026.05.06 08:48:36.703793 [ 32026 ] {} <Debug> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Creating shared ID for table test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768)
2026.05.06 08:48:36.703733 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:36.704029 [ 32244 ] {} <Test> KeeperLogStore: Flushing 5 logs
2026.05.06 08:48:36.704169 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:36.704247 [ 32038 ] {} <Debug> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Initializing table shared ID with a55ce901-4bf5-4668-b420-14c5d0a694bf
2026.05.06 08:48:36.704641 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 5, bytes: 500
2026.05.06 08:48:36.704662 [ 31937 ] {} <Debug> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Creating shared ID for table shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7)
2026.05.06 08:48:36.705338 [ 32244 ] {} <Test> KeeperLogStore: Flushing 5 logs
2026.05.06 08:48:36.705339 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Initializing table shared ID with 0d2b666f-a2a0-48ee-95c0-9c76f867f0f1
2026.05.06 08:48:36.705508 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 10, bytes: 1157
2026.05.06 08:48:36.706002 [ 32049 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Creating shared ID for table shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f)
2026.05.06 08:48:36.705473 [ 32063 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeCleanupThread): Checking 2 blocks (2 are not cached) to clear old ones from ZooKeeper., path is /clickhouse/tables/01414_mutations_and_errors_zookeeper_test_1/mutation_table/blocks
2026.05.06 08:48:36.707087 [ 32061 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.707418 [ 32026 ] {} <Debug> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Initializing table shared ID with 2edd1075-136c-4caa-a6fc-8fb7a8cd3845
2026.05.06 08:48:36.707395 [ 32244 ] {} <Test> KeeperLogStore: Flushing 10 logs
2026.05.06 08:48:36.707576 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 14, bytes: 1727
2026.05.06 08:48:36.708057 [ 32108 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Checking 2 blocks (2 are not cached) to clear old ones from ZooKeeper., path is /test/02485/test_13/rmt/blocks
2026.05.06 08:48:36.708254 [ 31937 ] {} <Debug> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Initializing table shared ID with 703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7
2026.05.06 08:48:36.709121 [ 32244 ] {} <Test> KeeperLogStore: Flushing 14 logs
2026.05.06 08:48:36.709311 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 17, bytes: 1795
2026.05.06 08:48:36.710920 [ 32244 ] {} <Test> KeeperLogStore: Flushing 17 logs
2026.05.06 08:48:36.711453 [ 32049 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Initializing table shared ID with a55ce901-4bf5-4668-b420-14c5d0a694bf
2026.05.06 08:48:36.711669 [ 31910 ] {} <Information> test_1.t (ReplicatedMergeTreeQueue): Loading 1 mutation entries: 0000000000 - 0000000000
2026.05.06 08:48:36.711911 [ 32445 ] {} <Trace> DDLWorker(test_11): scheduleTasks: initialized=true, size_before_filtering=1, queue_size=1, entries=query-0000000001..query-0000000001, first_failed_task_name=none, current_tasks_size=0, last_current_task=none, last_skipped_entry_name=query-0000000001
2026.05.06 08:48:36.711911 [ 32470 ] {} <Trace> DDLWorker(rdb_test_13_2): scheduleTasks: initialized=true, size_before_filtering=3, queue_size=3, entries=query-0000000001..query-0000000003, first_failed_task_name=none, current_tasks_size=0, last_current_task=none, last_skipped_entry_name=query-0000000001
2026.05.06 08:48:36.711942 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 13, bytes: 1619
2026.05.06 08:48:36.712053 [ 32468 ] {} <Trace> DDLWorker(rdb_test_13_3): scheduleTasks: initialized=true, size_before_filtering=3, queue_size=3, entries=query-0000000001..query-0000000003, first_failed_task_name=none, current_tasks_size=0, last_current_task=none, last_skipped_entry_name=query-0000000001
2026.05.06 08:48:36.712094 [ 32445 ] {} <Debug> DDLWorker(test_11): No tasks to schedule
2026.05.06 08:48:36.712062 [ 32440 ] {} <Trace> DDLWorker(test_5): scheduleTasks: initialized=true, size_before_filtering=1, queue_size=1, entries=query-0000000001..query-0000000001, first_failed_task_name=none, current_tasks_size=0, last_current_task=none, last_skipped_entry_name=query-0000000001
2026.05.06 08:48:36.712201 [ 32440 ] {} <Debug> DDLWorker(test_5): No tasks to schedule
2026.05.06 08:48:36.712201 [ 32445 ] {} <Debug> DDLWorker(test_11): Waiting for queue updates
2026.05.06 08:48:36.712254 [ 32440 ] {} <Debug> DDLWorker(test_5): Waiting for queue updates
2026.05.06 08:48:36.712211 [ 32468 ] {} <Debug> DDLWorker(rdb_test_13_3): Will schedule 2 tasks starting from query-0000000002
2026.05.06 08:48:36.712210 [ 32470 ] {} <Debug> DDLWorker(rdb_test_13_2): Will schedule 2 tasks starting from query-0000000002
2026.05.06 08:48:36.712403 [ 32470 ] {} <Trace> DDLWorker(rdb_test_13_2): Checking task query-0000000002
2026.05.06 08:48:36.712404 [ 32468 ] {} <Trace> DDLWorker(rdb_test_13_3): Checking task query-0000000002
2026.05.06 08:48:36.713259 [ 32244 ] {} <Test> KeeperLogStore: Flushing 13 logs
2026.05.06 08:48:36.713474 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 11, bytes: 1288
2026.05.06 08:48:36.714698 [ 32244 ] {} <Test> KeeperLogStore: Flushing 11 logs
2026.05.06 08:48:36.714730 [ 31911 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Selected 3 parts from all_0_0_0 to all_2_2_0
2026.05.06 08:48:36.714712 [ 32070 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.715328 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 2, bytes: 261
2026.05.06 08:48:36.715634 [ 32244 ] {} <Test> KeeperLogStore: Flushing 2 logs
2026.05.06 08:48:36.715962 [ 31910 ] {} <Trace> test_1.t (ReplicatedMergeTreeQueue): Adding mutation 0000000000 for 1 partitions (data versions: all = 1; )
2026.05.06 08:48:36.716158 [ 32104 ] {} <Debug> test_1.test_without_merge (ReplicatedMergeTreeQueue): Pulling 1 entries to queue: log-0000000003 - log-0000000003
2026.05.06 08:48:36.716267 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:36.716268 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:36.716445 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 127
2026.05.06 08:48:36.716553 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Creating shared ID for table test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1)
2026.05.06 08:48:36.716573 [ 32038 ] {} <Debug> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Creating shared ID for table shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf)
2026.05.06 08:48:36.716713 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.716774 [ 32038 ] {} <Information> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Shared ID already set to a55ce901-4bf5-4668-b420-14c5d0a694bf
2026.05.06 08:48:36.716772 [ 32021 ] {} <Information> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Shared ID already set to 0d2b666f-a2a0-48ee-95c0-9c76f867f0f1
2026.05.06 08:48:36.716778 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 126
2026.05.06 08:48:36.717130 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.717502 [ 32026 ] {} <Debug> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Creating shared ID for table test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768)
2026.05.06 08:48:36.717588 [ 32026 ] {} <Information> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Shared ID already set to 2edd1075-136c-4caa-a6fc-8fb7a8cd3845
2026.05.06 08:48:36.717757 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 343
2026.05.06 08:48:36.718005 [ 31937 ] {} <Debug> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Creating shared ID for table shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7)
2026.05.06 08:48:36.718057 [ 31937 ] {} <Information> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Shared ID already set to 703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7
2026.05.06 08:48:36.718156 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.718791 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 453
2026.05.06 08:48:36.718791 [ 31911 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Created log entry /clickhouse/tables/test_1/insert_dedup_token/log/log-0000000003 for merge all_0_2_1
2026.05.06 08:48:36.718825 [ 32038 ] {} <Information> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Became leader
2026.05.06 08:48:36.718863 [ 32021 ] {} <Information> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Became leader
2026.05.06 08:48:36.718932 [ 32049 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Creating shared ID for table shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f)
2026.05.06 08:48:36.718975 [ 32049 ] {} <Information> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Shared ID already set to a55ce901-4bf5-4668-b420-14c5d0a694bf
2026.05.06 08:48:36.718995 [ 32026 ] {} <Information> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Became leader
2026.05.06 08:48:36.719129 [ 32038 ] {} <Trace> shard_0.to (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.719151 [ 32021 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.719196 [ 32038 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Trying to startup table from right now
2026.05.06 08:48:36.719199 [ 32021 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Trying to startup table from right now
2026.05.06 08:48:36.719196 [ 32026 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.719292 [ 32038 ] {} <Debug> shard_0.to (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.719320 [ 32026 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Trying to startup table from right now
2026.05.06 08:48:36.719330 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.719338 [ 32038 ] {} <Debug> shard_0.to (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.719359 [ 32026 ] {} <Debug> test_13.rmt2 (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.719378 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.719368 [ 31911 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Selected 3 parts from all_0_0_0 to all_2_2_0
2026.05.06 08:48:36.719400 [ 32026 ] {} <Debug> test_13.rmt2 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.719546 [ 31911 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Checked 1 partitions, found 1 partitions with parts that may be merged: [all] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:36.719584 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.719727 [ 31910 ] {} <Test> test_1.t (ReplicatedMergeTreeQueue): Removing part all_0_0_1_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:36.720165 [ 32104 ] {} <Trace> test_1.test_without_merge (ReplicatedMergeTreeQueue): Insert entry queue-0000000003 to queue with type MERGE_PARTS with virtual parts [all_0_3_1]
2026.05.06 08:48:36.720194 [ 31937 ] {} <Information> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Became leader
2026.05.06 08:48:36.720253 [ 31937 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.720303 [ 31937 ] {} <Trace> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Trying to startup table from right now
2026.05.06 08:48:36.720290 [ 32104 ] {} <Test> test_1.test_without_merge (ReplicatedMergeTreeQueue): Adding part all_0_3_1 to mutations
2026.05.06 08:48:36.720344 [ 31937 ] {} <Debug> shard_0.from_1 (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.720393 [ 31937 ] {} <Debug> shard_0.from_1 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.720393 [ 32104 ] {} <Debug> test_1.test_without_merge (ReplicatedMergeTreeQueue): Pulled 1 entries to queue.
2026.05.06 08:48:36.720522 [ 32104 ] {} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Updating strategy picker state
2026.05.06 08:48:36.720588 [ 32049 ] {} <Information> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Became leader
2026.05.06 08:48:36.720664 [ 32049 ] {} <Trace> shard_1.to (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: false
2026.05.06 08:48:36.720714 [ 32049 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Trying to startup table from right now
2026.05.06 08:48:36.720737 [ 32049 ] {} <Debug> shard_1.to (ReplicatedMergeTreeRestartingThread): Activating replica.
2026.05.06 08:48:36.720765 [ 32049 ] {} <Debug> shard_1.to (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:48:36.721192 [ 32102 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeCleanupThread): Checking 11 blocks (11 are not cached) to clear old ones from ZooKeeper., path is /test/01417_freeze_partition_verbose_zookeeper_test_13/table_for_freeze_replicated/blocks
2026.05.06 08:48:36.721332 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:36.721359 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:36.722218 [ 32119 ] {} <Trace> test_1.t (ReplicatedMergeTreeQueue): Marking mutation 0000000000 done because it is <= mutation_pointer (0000000000)
2026.05.06 08:48:36.722300 [ 32123 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Found 2 old parts to remove. Parts: [all_0_0_0, all_0_0_1]
2026.05.06 08:48:36.722502 [ 31910 ] {} <Debug> test_1.t (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:36.722441 [ 32127 ] {} <Test> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:36.722445 [ 32107 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.722626 [ 31910 ] {} <Information> test_1.t (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:36.722764 [ 32467 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.t' with status OK
2026.05.06 08:48:36.722788 [ 32127 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:36.722753 [ 32123 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Removing 2 old parts from ZooKeeper
2026.05.06 08:48:36.722837 [ 32467 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:36.722913 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 64
2026.05.06 08:48:36.723187 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.723191 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:36.723191 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:36.723335 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:36.723335 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:36.723646 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 2, bytes: 400
2026.05.06 08:48:36.723921 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:36.723921 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:36.724310 [ 32244 ] {} <Test> KeeperLogStore: Flushing 2 logs
2026.05.06 08:48:36.724977 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 3, bytes: 692
2026.05.06 08:48:36.725193 [ 32281 ] {} <Trace> 920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1 (MergeFromLogEntryTask): Executing log entry to merge parts all_0_0_0, all_2_2_0, all_3_3_0 to all_0_3_1
2026.05.06 08:48:36.725292 [ 32118 ] {} <Debug> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Pulling 1 entries to queue: log-0000000003 - log-0000000003
2026.05.06 08:48:36.725654 [ 32281 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:36.725788 [ 32268 ] {} <Test> Poco: Queue size: 0, current threads: 0, threads in pool: 3, current connections: 0
2026.05.06 08:48:36.725907 [ 32268 ] {} <Test> Poco: Queue size: 0, current threads: 1, threads in pool: 3, current connections: 0
2026.05.06 08:48:36.725885 [ 32281 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:36.726004 [ 32244 ] {} <Test> KeeperLogStore: Flushing 3 logs
2026.05.06 08:48:36.726511 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 48000 rows, containing 3 columns (3 merged, 0 gathered) in 0.061779197 sec., 776960.5681342863 rows/sec., 19.27 MiB/sec.
2026.05.06 08:48:36.726778 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 290
2026.05.06 08:48:36.726809 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Debug> MergeTask::PrepareStage: Merging 3 parts: from all_0_0_0 to all_3_3_0 into Wide with storage Full
2026.05.06 08:48:36.726954 [ 32021 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.726990 [ 32468 ] {} <Trace> DatabaseReplicated (rdb_test_13_3): Got a list of hosts after 1 iterations. All hosts: [s9|r9, s2|r1, s1|r2], filtered: [s1|r2, s2|r1, s9|r9], ids: [e033c3a7a850:9000:4209e818-5442-4aed-bb4d-fa531de9528d, e033c3a7a850:9000:8b9134a0-8004-4d23-9dc7-d0d6beea89b2, e033c3a7a850:9000:8700d72c-0d5c-4681-aa98-ec5d5b057b3d]
2026.05.06 08:48:36.726991 [ 32470 ] {} <Trace> DatabaseReplicated (rdb_test_13_2): Got a list of hosts after 1 iterations. All hosts: [s9|r9, s2|r1, s1|r2], filtered: [s1|r2, s2|r1, s9|r9], ids: [e033c3a7a850:9000:4209e818-5442-4aed-bb4d-fa531de9528d, e033c3a7a850:9000:8b9134a0-8004-4d23-9dc7-d0d6beea89b2, e033c3a7a850:9000:8700d72c-0d5c-4681-aa98-ec5d5b057b3d]
2026.05.06 08:48:36.727160 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.727200 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 324
2026.05.06 08:48:36.727663 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Trace> MergedBlockOutputStream: filled checksums all_1_48_4 (state Temporary)
2026.05.06 08:48:36.727671 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.727921 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:36.727990 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 2, bytes: 148
2026.05.06 08:48:36.727963 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:36.728131 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Debug> MergeTask::PrepareStage: Will merge parts reading files in O_DIRECT
2026.05.06 08:48:36.728146 [ 32468 ] {} <Debug> DDLWorker(rdb_test_13_3): Will not execute task query-0000000002: Entry query-0000000002 is a dummy task
2026.05.06 08:48:36.728146 [ 32470 ] {} <Debug> DDLWorker(rdb_test_13_2): Will not execute task query-0000000002: Entry query-0000000002 is a dummy task
2026.05.06 08:48:36.728321 [ 32038 ] {} <Trace> shard_0.to (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.728332 [ 32470 ] {} <Trace> DDLWorker(rdb_test_13_2): Checking task query-0000000003
2026.05.06 08:48:36.728331 [ 32468 ] {} <Trace> DDLWorker(rdb_test_13_3): Checking task query-0000000003
2026.05.06 08:48:36.728434 [ 32244 ] {} <Test> KeeperLogStore: Flushing 2 logs
2026.05.06 08:48:36.728466 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_0_0_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.728624 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): Renaming temporary part tmp_merge_all_1_48_4 to all_1_48_4 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:36.728834 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Test> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2): preparePartForCommit: inserting all_1_48_4 (state PreActive) into data_parts_indexes
2026.05.06 08:48:36.728742 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:36.728931 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Trace> test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2) (MergerMutator): Merged 4 parts: [all_1_40_3, all_48_48_0] -> all_1_48_4
2026.05.06 08:48:36.729155 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 456
2026.05.06 08:48:36.729179 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2), part: all_1_40_3
2026.05.06 08:48:36.729188 [ 32049 ] {} <Trace> shard_1.to (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.729269 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2), part: all_41_46_1
2026.05.06 08:48:36.729333 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2), part: all_47_47_0
2026.05.06 08:48:36.729349 [ 32123 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Removed 2 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:36.729386 [ 32272 ] {27b45cb1-c21d-4330-8425-45169134d5c2::all_1_48_4} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.`03811_sparse_column_aggregation_with_sum` (27b45cb1-c21d-4330-8425-45169134d5c2), part: all_48_48_0
2026.05.06 08:48:36.729441 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.729454 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:36.729624 [ 32123 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Removing 2 parts from filesystem (serially): Parts: [all_0_0_0, all_0_0_1]
2026.05.06 08:48:36.729923 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.729886 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:36.730040 [ 32021 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Adding part all_2_2_0 to current and virtual parts
2026.05.06 08:48:36.730155 [ 32021 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Adding part all_0_0_0 to current and virtual parts
2026.05.06 08:48:36.730051 [ 32026 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.730226 [ 32021 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Adding part all_1_1_0 to current and virtual parts
2026.05.06 08:48:36.730264 [ 32118 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Insert entry queue-0000000003 to queue with type MERGE_PARTS with virtual parts [all_0_2_1]
2026.05.06 08:48:36.730294 [ 32021 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.730340 [ 32038 ] {} <Test> shard_0.to (ReplicatedMergeTreeQueue): Adding part all_0_3_1 to current and virtual parts
2026.05.06 08:48:36.730395 [ 32038 ] {} <Trace> shard_0.to (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.730389 [ 32118 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Adding part all_0_2_1 to mutations
2026.05.06 08:48:36.730307 [ 31937 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:48:36.730451 [ 32118 ] {} <Debug> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Pulled 1 entries to queue.
2026.05.06 08:48:36.730427 [ 32049 ] {} <Trace> shard_1.to (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.730511 [ 32118 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Updating strategy picker state
2026.05.06 08:48:36.730516 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/test_1/insert_dedup_token/replicas/r1/queue
2026.05.06 08:48:36.730560 [ 32049 ] {} <Debug> shard_1.to (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/to_test_uasu1ifx/replicas/1/queue
2026.05.06 08:48:36.730687 [ 32038 ] {} <Debug> shard_0.to (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/to_test_uasu1ifx/replicas/0/queue
2026.05.06 08:48:36.731024 [ 32123 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Part all_0_0_0 is not stored on zero-copy replicated disk, blobs can be removed
2026.05.06 08:48:36.731052 [ 32026 ] {} <Test> test_13.rmt2 (ReplicatedMergeTreeQueue): Adding part all_0_0_0_2 to current and virtual parts
2026.05.06 08:48:36.731030 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:36.731183 [ 32026 ] {} <Test> test_13.rmt2 (ReplicatedMergeTreeQueue): Adding part all_1_1_0_2 to current and virtual parts
2026.05.06 08:48:36.731260 [ 32026 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.731356 [ 31937 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:48:36.731637 [ 32026 ] {} <Debug> test_13.rmt2 (ReplicatedMergeTreeQueue): Loading queue from /test/02485/test_13/rmt/replicas/2/queue
2026.05.06 08:48:36.731667 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.731672 [ 31937 ] {} <Debug> shard_0.from_1 (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/from_1_test_uasu1ifx/replicas/0/queue
2026.05.06 08:48:36.731725 [ 32049 ] {} <Debug> shard_1.to (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.731725 [ 32038 ] {} <Debug> shard_0.to (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.732007 [ 32272 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 6.67 MiB.
2026.05.06 08:48:36.732024 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:36.732151 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:36.732222 [ 32026 ] {} <Debug> test_13.rmt2 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.732294 [ 32006 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.732537 [ 31937 ] {} <Debug> shard_0.from_1 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:48:36.732638 [ 32038 ] {} <Debug> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Updating strategy picker state
2026.05.06 08:48:36.732628 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Updating strategy picker state
2026.05.06 08:48:36.732715 [ 32049 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Updating strategy picker state
2026.05.06 08:48:36.733156 [ 32118 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Strategy picker state updated, current replica: 1, active replicas: [r1, r2]
2026.05.06 08:48:36.733511 [ 32026 ] {} <Debug> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Updating strategy picker state
2026.05.06 08:48:36.733670 [ 32273 ] {} <Trace> 0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1 (MergeFromLogEntryTask): Executing log entry to merge parts all_0_0_0, all_1_1_0, all_2_2_0 to all_0_2_1
2026.05.06 08:48:36.733718 [ 31937 ] {} <Debug> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Updating strategy picker state
2026.05.06 08:48:36.733888 [ 32273 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:36.733968 [ 32123 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Part all_0_0_1 is not stored on zero-copy replicated disk, blobs can be removed
2026.05.06 08:48:36.733981 [ 32273 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:36.734341 [ 32273 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Debug> MergeTask::PrepareStage: Merging 3 parts: from all_0_0_0 to all_2_2_0 into Compact with storage Full
2026.05.06 08:48:36.734588 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:36.734588 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:36.734803 [ 32273 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:36.735095 [ 32273 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Debug> MergeTask::PrepareStage: Will merge parts reading files in O_DIRECT
2026.05.06 08:48:36.735273 [ 32273 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_0_0_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.735505 [ 32038 ] {} <Debug> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Strategy picker state updated, current replica: 0, active replicas: [0, 1]
2026.05.06 08:48:36.735555 [ 32038 ] {} <Trace> shard_0.to (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:36.735793 [ 32127 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:36.736055 [ 32391 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.736085 [ 32273 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.736272 [ 32123 ] {} <Test> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): removePartsFinally: removing all_0_0_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:36.736327 [ 32362 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.736321 [ 32273 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:36.736397 [ 32123 ] {} <Test> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): removePartsFinally: removing all_0_0_1 (state Deleting) from data_parts_indexes
2026.05.06 08:48:36.736455 [ 32123 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Removing 2 parts from memory: Parts: [all_0_0_0, all_0_0_1]
2026.05.06 08:48:36.736574 [ 32384 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:36.736660 [ 32123 ] {} <Debug> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Removed 2 old parts
2026.05.06 08:48:36.737272 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:36.737374 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:36.737526 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:36.737566 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:36.737587 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:36.737682 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:36.737794 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:36.737828 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:36.743261 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:36.743264 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:36.743447 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:36.743450 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:36.745753 [ 32277 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 3 rows, containing 3 columns (3 merged, 0 gathered) in 0.011540084 sec., 259.96344567335905 rows/sec., 4.06 KiB/sec.
2026.05.06 08:48:36.745753 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 3 rows, containing 2 columns (2 merged, 0 gathered) in 0.019410324 sec., 154.55692547945105 rows/sec., 2.41 KiB/sec.
2026.05.06 08:48:36.746541 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Trace> MergedBlockOutputStream: filled checksums all_0_3_1 (state Temporary)
2026.05.06 08:48:36.746760 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Trace> MergedBlockOutputStream: filled checksums all_0_2_1 (state Temporary)
2026.05.06 08:48:36.748543 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Renaming temporary part tmp_merge_all_0_2_1 to all_0_2_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:36.749400 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Test> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): preparePartForCommit: inserting all_0_2_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:36.749621 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Merged 3 parts: [all_0_0_0, all_2_2_0] -> all_0_2_1
2026.05.06 08:48:36.749738 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Renaming temporary part tmp_merge_all_0_3_1 to all_0_3_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:36.749903 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Test> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): preparePartForCommit: inserting all_0_3_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:36.749960 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55) (MergerMutator): Merged 3 parts: [all_0_0_0, all_3_3_0] -> all_0_3_1
2026.05.06 08:48:36.751325 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Committing part all_0_3_1 to zookeeper
2026.05.06 08:48:36.751850 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Committing part all_0_2_1 to zookeeper
2026.05.06 08:48:43.436993 [ 32481 ] {} <Test> Poco: Queue size: 0, current threads: 1, threads in pool: 4, current connections: 0
2026.05.06 08:48:43.437555 [ 32447 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 2.21 GiB, peak 2.23 GiB, free memory in arenas 0.00 B, will set to 2.32 GiB (RSS), difference: 115.23 MiB
2026.05.06 08:48:43.441102 [ 32427 ] {} <Trace> ExternalDictionariesLoader: Loading config file '9a4dceb0-e264-42cf-a806-eebbe724dabd'.
2026.05.06 08:48:43.449605 [ 31641 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:58926
2026.05.06 08:48:43.449178 [ 32359 ] {} <Trace> SystemLog (system.text_log): Flushing system log, 6870 entries to flush up to offset 6870
2026.05.06 08:48:43.441107 [ 32383 ] {} <Trace> SystemLog (system.zookeeper_log): Flushing system log, 3085 entries to flush up to offset 3085
2026.05.06 08:48:43.477716 [ 32360 ] {} <Trace> SystemLog (system.transactions_info_log): Flushing system log, 72 entries to flush up to offset 72
2026.05.06 08:48:43.475025 [ 31640 ] {} <Information> KeeperTCPHandler: Waiting for response to be ready for session 5 took 6714 ms
2026.05.06 08:48:43.447417 [ 32400 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushing system log, 576 entries to flush up to offset 576
2026.05.06 08:48:43.481627 [ 32385 ] {} <Trace> SystemLog (system.query_log): Flushing system log, 4 entries to flush up to offset 4
2026.05.06 08:48:43.481679 [ 32349 ] {} <Trace> SystemLog (system.metric_log): Flushing system log, 2 entries to flush up to offset 2
2026.05.06 08:48:43.485610 [ 32347 ] {} <Trace> SystemLog (system.error_log): Flushing system log, 3 entries to flush up to offset 3
2026.05.06 08:48:43.477903 [ 32373 ] {} <Trace> SystemLog (system.processors_profile_log): Flushing system log, 34 entries to flush up to offset 34
2026.05.06 08:48:43.485814 [ 32427 ] {} <Trace> ExternalDictionariesLoader: Loading config file 'a079a21a-41a1-4345-9b89-606f0b5dd6b8'.
2026.05.06 08:48:43.480446 [ 32375 ] {} <Trace> SystemLog (system.trace_log): Flushing system log, 78 entries to flush up to offset 78
2026.05.06 08:48:43.487540 [ 32399 ] {} <Trace> SystemLog (system.blob_storage_log): Flushing system log, 4 entries to flush up to offset 4
2026.05.06 08:48:43.478611 [ 32338 ] {} <Trace> SystemLog (system.part_log): Flushing system log, 7 entries to flush up to offset 7
2026.05.06 08:48:43.491570 [ 31640 ] {} <Information> KeeperTCPHandler: Total time to process a request in session 5 took too long (6755ms).
Request info: XID = 1471
OpNum = Exists
Additional info:
path = /clickhouse/tables/test_1/insert_dedup_token/replicas/r2/is_active
2026.05.06 08:48:43.492815 [ 32427 ] {} <Trace> ExternalDictionariesLoader: Loading config file '9000190c-31ea-4742-a77d-a0d20c9cb3fe'.
2026.05.06 08:48:43.493165 [ 32427 ] {} <Trace> ExternalDictionariesLoader: Loading config file '97f1625b-b420-40a0-ac0e-b1f4e14ed0d9'.
2026.05.06 08:48:43.493625 [ 31640 ] {} <Test> virtual Coordination::ZooKeeperRequest::~ZooKeeperRequest(): Processing of request xid=1471 took 6758 ms
2026.05.06 08:48:43.494071 [ 31640 ] {} <Test> virtual Coordination::ZooKeeperResponse::~ZooKeeperResponse(): Processing of response xid=1471 took 6758 ms
2026.05.06 08:48:43.494481 [ 31640 ] {} <Information> KeeperTCPHandler: Total time to process a request in session 5 took too long (6759ms).
Request info: XID = 1472
OpNum = Exists
Additional info:
path = /clickhouse/tables/to_test_uasu1ifx/replicas/1/is_active
2026.05.06 08:48:43.494706 [ 31640 ] {} <Test> virtual Coordination::ZooKeeperRequest::~ZooKeeperRequest(): Processing of request xid=1472 took 6759 ms
2026.05.06 08:48:43.494799 [ 31640 ] {} <Test> virtual Coordination::ZooKeeperResponse::~ZooKeeperResponse(): Processing of response xid=1472 took 6759 ms
2026.05.06 08:48:43.495884 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:43.496517 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.497193 [ 32049 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Strategy picker state updated, current replica: 1, active replicas: [0, 1]
2026.05.06 08:48:43.497296 [ 32049 ] {} <Trace> shard_1.to (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:43.497295 [ 32470 ] {} <Trace> DatabaseReplicated (rdb_test_13_2): Got a list of hosts after 1 iterations. All hosts: [s1|r2, s2|r1, s9|r9], filtered: [s1|r2, s2|r1, s9|r9], ids: [e033c3a7a850:9000:4209e818-5442-4aed-bb4d-fa531de9528d, e033c3a7a850:9000:8b9134a0-8004-4d23-9dc7-d0d6beea89b2, e033c3a7a850:9000:8700d72c-0d5c-4681-aa98-ec5d5b057b3d]
2026.05.06 08:48:43.497303 [ 32468 ] {} <Trace> DatabaseReplicated (rdb_test_13_3): Got a list of hosts after 1 iterations. All hosts: [s1|r2, s2|r1, s9|r9], filtered: [s1|r2, s2|r1, s9|r9], ids: [e033c3a7a850:9000:4209e818-5442-4aed-bb4d-fa531de9528d, e033c3a7a850:9000:8b9134a0-8004-4d23-9dc7-d0d6beea89b2, e033c3a7a850:9000:8700d72c-0d5c-4681-aa98-ec5d5b057b3d]
2026.05.06 08:48:43.497354 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Strategy picker state updated, current replica: 0, active replicas: [r1, r2]
2026.05.06 08:48:43.497448 [ 32021 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:43.497603 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:43.497964 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.499108 [ 31641 ] {} <Debug> TCPHandler: Connected ClickHouse client version 24.8.0, revision: 54472, user: default.
2026.05.06 08:48:43.499308 [ 31641 ] {} <Debug> TCP-Session-1d0fe026-ad5d-425a-979a-cb87d39af877: Authenticating user 'default' from [::1]:58926
2026.05.06 08:48:43.500176 [ 32026 ] {} <Debug> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Strategy picker state updated, current replica: 1, active replicas: [1, 2]
2026.05.06 08:48:43.500230 [ 31937 ] {} <Debug> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Strategy picker state updated, current replica: 0, active replicas: [0, 1]
2026.05.06 08:48:43.500282 [ 32026 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:43.500323 [ 31937 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:48:43.501527 [ 32468 ] {} <Debug> DDLWorker(rdb_test_13_3): Will not execute task query-0000000003: Entry query-0000000003 is a dummy task
2026.05.06 08:48:43.501542 [ 31641 ] {} <Debug> TCP-Session-1d0fe026-ad5d-425a-979a-cb87d39af877: 1d0fe026-ad5d-425a-979a-cb87d39af877 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:43.501837 [ 32470 ] {} <Debug> DDLWorker(rdb_test_13_2): Will not execute task query-0000000003: Entry query-0000000003 is a dummy task
2026.05.06 08:48:43.502218 [ 31641 ] {} <Debug> TCP-Session-1d0fe026-ad5d-425a-979a-cb87d39af877: Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:43.502225 [ 32427 ] {} <Trace> ExternalDictionariesLoader: Loading config file 'b4c7de75-3d1a-4b04-822d-0177baf9641c'.
2026.05.06 08:48:43.502374 [ 32427 ] {} <Trace> ExternalDictionariesLoader: Loading config file 'e1157c8f-9651-4b58-824a-103691fe26b6'.
2026.05.06 08:48:43.502480 [ 32427 ] {} <Trace> ExternalDictionariesLoader: Loading config file '701b3c4b-fac3-4317-a21a-efc26faff67c'.
2026.05.06 08:48:43.502605 [ 32427 ] {} <Trace> ExternalDictionariesLoader: Loading config file '7cead437-bf5d-4424-8ba0-3977fac8fce1'.
2026.05.06 08:48:43.503792 [ 32468 ] {} <Debug> DDLWorker(rdb_test_13_3): Waiting for queue updates
2026.05.06 08:48:43.503961 [ 32427 ] {} <Trace> ExternalDictionariesLoader: Loading config file 'e56ec67b-a9c3-4381-902c-079b6b746314'.
2026.05.06 08:48:43.503958 [ 32470 ] {} <Debug> DDLWorker(rdb_test_13_2): Waiting for queue updates
2026.05.06 08:48:43.504196 [ 32427 ] {} <Trace> ExternalDictionariesLoader: Loading config file '71161200-93e8-40fd-b11d-983febc99551'.
2026.05.06 08:48:43.504429 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:43.504429 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:43.508177 [ 32311 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:43.508910 [ 32123 ] {} <Trace> test_1.t (ReplicatedMergeTreeCleanupThread): Checking 1 blocks (1 are not cached) to clear old ones from ZooKeeper., path is /clickhouse/test/test_1/test_02477/blocks
2026.05.06 08:48:43.516871 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:43.516873 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:43.517343 [ 31911 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:43.517550 [ 31911 ] {} <Trace> test_1.insert_dedup_token2 (StorageReplicatedMergeTree::mergeSelectingTask): Execution took 6798 ms.
2026.05.06 08:48:43.518206 [ 32336 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/test02676_without_merge/replicas/node/parts/all_0_3_1
2026.05.06 08:48:43.518578 [ 32336 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/test02676_without_merge/replicas/node/parts/all_0_3_1
2026.05.06 08:48:43.518815 [ 32123 ] {} <Trace> test_1.t (ReplicatedMergeTreeCleanupThread): Execution took 6796 ms.
2026.05.06 08:48:43.525533 [ 32336 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/insert_dedup_token/replicas/r1/parts/all_0_2_1
2026.05.06 08:48:43.525519 [ 32311 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: ::1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:43 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE72AB412057D
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:43.526252 [ 32311 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:43.526418 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 548
2026.05.06 08:48:43.518708 [ 32062 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 6816ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:43.526615 [ 32239 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/test02676_without_merge/replicas/node/parts/all_0_3_1
2026.05.06 08:48:43.526847 [ 32239 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/test02676_without_merge/replicas/node/parts/all_0_3_1
2026.05.06 08:48:43.527596 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.527893 [ 32373 ] {} <Debug> SystemLog (system.processors_profile_log): Will use existing table system.processors_profile_log for ProcessorsProfileLog
2026.05.06 08:48:43.525581 [ 32336 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/insert_dedup_token/replicas/r2/parts/all_0_2_1
2026.05.06 08:48:43.518702 [ 32060 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 6851ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:43.528691 [ 32336 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/insert_dedup_token/replicas/r2/parts/all_0_2_1
2026.05.06 08:48:43.529935 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Debug> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Part all_0_3_1 committed to zookeeper
2026.05.06 08:48:43.530449 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 742
2026.05.06 08:48:43.530552 [ 32239 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/insert_dedup_token/replicas/r1/parts/all_0_2_1
2026.05.06 08:48:43.532516 [ 32239 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/insert_dedup_token/replicas/r2/parts/all_0_2_1
2026.05.06 08:48:43.532582 [ 32239 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/insert_dedup_token/replicas/r2/parts/all_0_2_1
2026.05.06 08:48:43.532585 [ 32359 ] {} <Debug> SystemLog (system.text_log): Will use existing table system.text_log for TextLog
2026.05.06 08:48:43.533341 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.533330 [ 32400 ] {} <Debug> SystemLog (system.asynchronous_metric_log): Will use existing table system.asynchronous_metric_log for AsynchronousMetricLog
2026.05.06 08:48:43.534442 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55), part: all_0_0_0
2026.05.06 08:48:43.534565 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Part all_0_2_1 committed to zookeeper
2026.05.06 08:48:43.534688 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55), part: all_2_2_0
2026.05.06 08:48:43.534839 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55), part: all_3_3_0
2026.05.06 08:48:43.534853 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355), part: all_0_0_0
2026.05.06 08:48:43.534942 [ 32360 ] {} <Debug> SystemLog (system.transactions_info_log): Will use existing table system.transactions_info_log for TransactionsInfoLog
2026.05.06 08:48:43.535014 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355), part: all_1_1_0
2026.05.06 08:48:43.535139 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355), part: all_2_2_0
2026.05.06 08:48:43.535989 [ 32347 ] {} <Debug> SystemLog (system.error_log): Will use existing table system.error_log for ErrorLog
2026.05.06 08:48:43.536333 [ 32026 ] {} <Information> test_13.rmt2 (ReplicatedMergeTreeQueue): Loading 1 mutation entries: 0000000000 - 0000000000
2026.05.06 08:48:43.536192 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Pulling 1 entries to queue: log-0000000003 - log-0000000003
2026.05.06 08:48:43.536117 [ 32049 ] {} <Debug> shard_1.to (ReplicatedMergeTreeQueue): Pulling 1 entries to queue: log-0000000000 - log-0000000000
2026.05.06 08:48:43.536749 [ 32375 ] {} <Debug> SystemLog (system.trace_log): Will use existing table system.trace_log for TraceLog
2026.05.06 08:48:43.537011 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Test> test_1.test_without_merge (ReplicatedMergeTreeQueue): Removing successful entry queue-0000000003 from queue with type MERGE_PARTS with virtual parts [all_0_3_1]
2026.05.06 08:48:43.537012 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Removing successful entry queue-0000000003 from queue with type MERGE_PARTS with virtual parts [all_0_2_1]
2026.05.06 08:48:43.537376 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Test> test_1.test_without_merge (ReplicatedMergeTreeQueue): Adding parts [all_0_3_1] to current parts
2026.05.06 08:48:43.537377 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Adding parts [all_0_2_1] to current parts
2026.05.06 08:48:43.538005 [ 32280 ] {0486a7ae-5268-42e9-bce5-dd91cd54e355::all_0_2_1} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Removing part all_0_2_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.538005 [ 32281 ] {920b4c2c-6de3-4d9a-aa58-0f0e78c16a55::all_0_3_1} <Test> test_1.test_without_merge (ReplicatedMergeTreeQueue): Removing part all_0_3_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.539738 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 87
2026.05.06 08:48:43.540094 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.540604 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 94
2026.05.06 08:48:43.540864 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.541116 [ 32026 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeQueue): Adding mutation 0000000000 for 1 partitions (data versions: all = 2; )
2026.05.06 08:48:43.541561 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:43.541754 [ 32038 ] {} <Test> shard_0.to (ReplicatedMergeTreeQueue): Removing part all_0_3_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.541843 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:43.542062 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:43.542320 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.542598 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.542803 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:43.542966 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:43.542978 [ 32280 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 4.15 MiB.
2026.05.06 08:48:43.543190 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:43.543472 [ 32059 ] {} <Test> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:43.541562 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:43.543806 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:43.543973 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:43.544056 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 456
2026.05.06 08:48:43.544269 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.544472 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.544649 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:43.544755 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:43.544738 [ 32059 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:43.544955 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:43.544985 [ 32281 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 5.29 MiB.
2026.05.06 08:48:43.544984 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.544996 [ 32059 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:43.545357 [ 32059 ] {} <Test> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:43.545478 [ 32059 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:43.545635 [ 32059 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:43.545691 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 618
2026.05.06 08:48:43.546169 [ 32021 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Insert entry queue-0000000003 to queue with type MERGE_PARTS with virtual parts [all_0_2_1]
2026.05.06 08:48:43.546518 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.546791 [ 32021 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Adding part all_0_2_1 to mutations
2026.05.06 08:48:43.547927 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Pulled 1 entries to queue.
2026.05.06 08:48:43.548206 [ 31641 ] {} <Trace> TCP-Session-1d0fe026-ad5d-425a-979a-cb87d39af877: 1d0fe026-ad5d-425a-979a-cb87d39af877 Creating query context from session context, user_id: 94309d50-4f52-5250-31bd-74fecac179db, parent context user: default
2026.05.06 08:48:43.548284 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Updating strategy picker state
2026.05.06 08:48:43.548578 [ 32049 ] {} <Trace> shard_1.to (ReplicatedMergeTreeQueue): Insert entry queue-0000000000 to queue with type REPLACE_RANGE with virtual parts [all_0_0_0, all_1_1_0]
2026.05.06 08:48:43.548646 [ 32049 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Adding part all_0_0_0 to mutations
2026.05.06 08:48:43.548683 [ 32049 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Adding part all_1_1_0 to mutations
2026.05.06 08:48:43.548723 [ 32049 ] {} <Debug> shard_1.to (ReplicatedMergeTreeQueue): Pulled 1 entries to queue.
2026.05.06 08:48:43.548761 [ 32049 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Updating strategy picker state
2026.05.06 08:48:43.548769 [ 32399 ] {} <Debug> SystemLog (system.blob_storage_log): Will use existing table system.blob_storage_log for BlobStorageLog
2026.05.06 08:48:43.549238 [ 32338 ] {} <Debug> SystemLog (system.part_log): Will use existing table system.part_log for PartLog
2026.05.06 08:48:43.549540 [ 32385 ] {} <Debug> SystemLog (system.query_log): Will use existing table system.query_log for QueryLog
2026.05.06 08:48:43.549644 [ 31641 ] {0db48258-83bf-447d-b550-aae5c606881e} <Debug> executeQuery: (from [::1]:58926) SELECT 'Connected to clickhouse-server after attaching gdb' (stage: Complete)
2026.05.06 08:48:43.549766 [ 32026 ] {} <Test> test_13.rmt2 (ReplicatedMergeTreeQueue): Removing part all_0_0_0_2 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.549815 [ 32026 ] {} <Test> test_13.rmt2 (ReplicatedMergeTreeQueue): Removing part all_1_1_0_2 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.549875 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:43.550085 [ 32038 ] {} <Debug> shard_0.to (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:43.550197 [ 32038 ] {} <Information> shard_0.to (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:43.549876 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:43.550281 [ 32038 ] {} <Trace> shard_0.to (ReplicatedMergeTreeAttachThread): Execution took 6884 ms.
2026.05.06 08:48:43.550350 [ 32436 ] {} <Debug> AsyncLoader: Finish load job 'startup table shard_0.to' with status OK
2026.05.06 08:48:43.550436 [ 32436 ] {} <Information> AsyncLoader: Processed: 97.2%
2026.05.06 08:48:43.550500 [ 32436 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.550922 [ 32026 ] {} <Debug> test_13.rmt2 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:43.550995 [ 32026 ] {} <Information> test_13.rmt2 (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:43.551049 [ 32026 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeAttachThread): Execution took 6890 ms.
2026.05.06 08:48:43.551099 [ 32459 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_13.rmt2' with status OK
2026.05.06 08:48:43.551157 [ 32459 ] {} <Debug> AsyncLoader: Spawn loader worker #5 in BackgrndStartup
2026.05.06 08:48:43.551278 [ 32459 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_13' in BackgrndStartup
2026.05.06 08:48:43.551340 [ 32459 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_13' with status OK
2026.05.06 08:48:43.551374 [ 32459 ] {} <Debug> AsyncLoader: Spawn loader worker #6 in BackgrndStartup
2026.05.06 08:48:43.551427 [ 32459 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_13' in BackgrndStartup
2026.05.06 08:48:43.551462 [ 32459 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_13' with status OK
2026.05.06 08:48:43.551504 [ 32459 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.551582 [ 32475 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.550286 [ 32052 ] {} <Test> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:43.551655 [ 32448 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.550981 [ 32043 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeQueue): Marking mutation 0000000000 done because it is <= mutation_pointer (0000000000)
2026.05.06 08:48:43.552541 [ 31641 ] {0db48258-83bf-447d-b550-aae5c606881e} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:43.551743 [ 32052 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:43.556805 [ 32052 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:43.557506 [ 32047 ] {} <Test> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:43.557646 [ 32047 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:43.557727 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:43.557727 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:43.557968 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:43.557970 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:43.558057 [ 31641 ] {0db48258-83bf-447d-b550-aae5c606881e} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:43.558343 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.558345 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.558620 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.558623 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.558660 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Strategy picker state updated, current replica: 0, active replicas: [r1, r2]
2026.05.06 08:48:43.558867 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:43.558868 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:43.559021 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:43.559022 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:43.559287 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:43.559288 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:43.559978 [ 32041 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Checking 2 blocks (2 are not cached) to clear old ones from ZooKeeper., path is /test/02485/test_13/rmt/blocks
2026.05.06 08:48:43.560018 [ 32049 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Strategy picker state updated, current replica: 1, active replicas: [0, 1]
2026.05.06 08:48:43.560342 [ 32049 ] {} <Debug> shard_1.to (ReplicatedMergeTreeQueue): Pulling 2 entries to queue: log-0000000001 - log-0000000002
2026.05.06 08:48:43.560474 [ 31937 ] {} <Debug> shard_0.from_1 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:43.560589 [ 32034 ] {} <Trace> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:43.560607 [ 31937 ] {} <Information> shard_0.from_1 (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:43.560742 [ 31937 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeAttachThread): Execution took 6893 ms.
2026.05.06 08:48:43.560857 [ 32442 ] {} <Debug> AsyncLoader: Finish load job 'startup table shard_0.from_1' with status OK
2026.05.06 08:48:43.560998 [ 32442 ] {} <Debug> AsyncLoader: Spawn loader worker #4 in BackgrndStartup
2026.05.06 08:48:43.561239 [ 32442 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database shard_0' in BackgrndStartup
2026.05.06 08:48:43.561370 [ 32442 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database shard_0' with status OK
2026.05.06 08:48:43.561461 [ 32442 ] {} <Debug> AsyncLoader: Spawn loader worker #5 in BackgrndStartup
2026.05.06 08:48:43.561608 [ 32442 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database shard_0' in BackgrndStartup
2026.05.06 08:48:43.561637 [ 32347 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:43.561685 [ 32442 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database shard_0' with status OK
2026.05.06 08:48:43.561754 [ 32442 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.561848 [ 32428 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.561952 [ 32430 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.562168 [ 32021 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Removing part all_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.562321 [ 32021 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Removing part all_1_1_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.562411 [ 32021 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Removing part all_2_2_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.562432 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:43.562551 [ 32383 ] {} <Debug> SystemLog (system.zookeeper_log): Will use existing table system.zookeeper_log for ZooKeeperLog
2026.05.06 08:48:43.562923 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 915
2026.05.06 08:48:43.562979 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:43.563511 [ 32360 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:43.563797 [ 31641 ] {0db48258-83bf-447d-b550-aae5c606881e} <Debug> executeQuery: Read 1 rows, 1.00 B in 0.01457 sec., 68.63417982155113 rows/sec., 68.63 B/sec.
2026.05.06 08:48:43.564745 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:43.564745 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:43.564952 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.564978 [ 32375 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:43.565116 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:43.565116 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:43.565411 [ 31641 ] {0db48258-83bf-447d-b550-aae5c606881e} <Debug> TCPHandler: Processed in 0.017949402 sec.
2026.05.06 08:48:43.565620 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.565620 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.565837 [ 32285 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Updating size with -1, current is 1281
2026.05.06 08:48:43.565928 [ 32285 ] {} <Test> LRUFileCachePriority(s3_cache, probationary): Removed entry from LRU queue, key: e975169734962c94d801165a31501fba, offset: 0, size: 1
2026.05.06 08:48:43.566022 [ 32285 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Updating size with 1, current is 1491
2026.05.06 08:48:43.566113 [ 32285 ] {} <Test> LRUFileCachePriority(s3_cache, protected): Added entry into LRU queue, key: e975169734962c94d801165a31501fba, offset: 0, size: 1
2026.05.06 08:48:43.566237 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.566386 [ 32049 ] {} <Trace> shard_1.to (ReplicatedMergeTreeQueue): Insert entry queue-0000000001 to queue with type REPLACE_RANGE with virtual parts [all_2_2_0, all_3_3_0]
2026.05.06 08:48:43.566499 [ 32049 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Adding part all_2_2_0 to mutations
2026.05.06 08:48:43.566564 [ 32049 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Adding part all_3_3_0 to mutations
2026.05.06 08:48:43.566655 [ 32049 ] {} <Trace> shard_1.to (ReplicatedMergeTreeQueue): Insert entry queue-0000000002 to queue with type MERGE_PARTS with virtual parts [all_0_3_1]
2026.05.06 08:48:43.566677 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:43.566724 [ 32049 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Adding part all_0_3_1 to mutations
2026.05.06 08:48:43.566771 [ 32049 ] {} <Debug> shard_1.to (ReplicatedMergeTreeQueue): Pulled 2 entries to queue.
2026.05.06 08:48:43.566798 [ 32286 ] {} <Trace> 0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1 (MergeFromLogEntryTask): Executing log entry to merge parts all_0_0_0, all_1_1_0, all_2_2_0 to all_0_2_1
2026.05.06 08:48:43.566826 [ 32027 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:43.566576 [ 32021 ] {} <Debug> test_1.insert_dedup_token1 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:43.566934 [ 32286 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.566956 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:43.567005 [ 32021 ] {} <Information> test_1.insert_dedup_token1 (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:43.567024 [ 32286 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.567125 [ 32021 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeAttachThread): Execution took 6908 ms.
2026.05.06 08:48:43.567177 [ 32344 ] {} <Debug> AsyncLoader: Finish load job 'startup table test_1.insert_dedup_token1' with status OK
2026.05.06 08:48:43.567262 [ 32344 ] {} <Debug> AsyncLoader: Spawn loader worker #3 in BackgrndStartup
2026.05.06 08:48:43.567275 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.567361 [ 32286 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Debug> MergeTask::PrepareStage: Merging 3 parts: from all_0_0_0 to all_2_2_0 into Compact with storage Full
2026.05.06 08:48:43.567394 [ 32344 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database test_1' in BackgrndStartup
2026.05.06 08:48:43.567411 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:43.567420 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:43.567453 [ 32344 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database test_1' with status OK
2026.05.06 08:48:43.567506 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:43.567665 [ 32344 ] {} <Debug> AsyncLoader: Spawn loader worker #4 in BackgrndStartup
2026.05.06 08:48:43.567699 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:43.567736 [ 32286 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:43.567764 [ 32373 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:43.567794 [ 32286 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Debug> MergeTask::PrepareStage: Will merge parts reading files in O_DIRECT
2026.05.06 08:48:43.567868 [ 32344 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database test_1' in BackgrndStartup
2026.05.06 08:48:43.567942 [ 32286 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_0_0_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:43.568011 [ 32344 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database test_1' with status OK
2026.05.06 08:48:43.568106 [ 32344 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.568256 [ 32461 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:43.568268 [ 32477 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.568288 [ 32286 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:43.568377 [ 32473 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.568405 [ 32347 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.568559 [ 32286 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:43.568630 [ 32393 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:43.568871 [ 32360 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.568871 [ 32347 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.568949 [ 32391 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:43.569178 [ 32360 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.566505 [ 32399 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:43.571301 [ 32022 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeCleanupThread): Checking 3 blocks (3 are not cached) to clear old ones from ZooKeeper., path is /clickhouse/tables/test_1/insert_dedup_token/blocks
2026.05.06 08:48:43.571562 [ 32047 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:43.572718 [ 32375 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.572992 [ 32049 ] {} <Debug> shard_1.to (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:48:43.573043 [ 32375 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.573066 [ 32049 ] {} <Information> shard_1.to (ReplicatedMergeTreeAttachThread): Table is initialized
2026.05.06 08:48:43.573136 [ 32049 ] {} <Trace> shard_1.to (ReplicatedMergeTreeAttachThread): Execution took 6900 ms.
2026.05.06 08:48:43.573204 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup table shard_1.to' with status OK
2026.05.06 08:48:43.573287 [ 32439 ] {} <Debug> AsyncLoader: Spawn loader worker #2 in BackgrndStartup
2026.05.06 08:48:43.573333 [ 32013 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:43.573422 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup Ordinary database shard_1' in BackgrndStartup
2026.05.06 08:48:43.573496 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup Ordinary database shard_1' with status OK
2026.05.06 08:48:43.573531 [ 32439 ] {} <Debug> AsyncLoader: Spawn loader worker #3 in BackgrndStartup
2026.05.06 08:48:43.573512 [ 32312 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Executing log entry queue-0000000000 to replace parts range all_0_0_0 with 2 parts from shard_0.from_0
2026.05.06 08:48:43.573590 [ 32439 ] {} <Debug> AsyncLoader: Execute load job 'startup Atomic database shard_1' in BackgrndStartup
2026.05.06 08:48:43.573628 [ 32439 ] {} <Debug> AsyncLoader: Finish load job 'startup Atomic database shard_1' with status OK
2026.05.06 08:48:43.573621 [ 32314 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Executing log entry queue-0000000001 to replace parts range all_0_0_0 with 2 parts from shard_0.from_1
2026.05.06 08:48:43.573663 [ 32439 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.573752 [ 32469 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.573859 [ 32472 ] {} <Debug> AsyncLoader: Stop worker in BackgrndStartup
2026.05.06 08:48:43.573905 [ 32472 ] {} <Debug> AsyncLoader: Change current priority: 2 -> none
2026.05.06 08:48:43.573735 [ 32016 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 for part all_0_3_1 because it is not disjoint with part all_0_0_0 that is currently executing.
2026.05.06 08:48:43.574172 [ 32314 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table shard_0.from_1': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:43.574751 [ 32314 ] {} <Debug> AsyncLoader: Prioritize load job 'load table shard_0.from_1': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:43.574844 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:43.575099 [ 32312 ] {} <Debug> AsyncLoader: Prioritize load job 'startup table shard_0.from_0': BackgrndStartup -> ForegroundLoad
2026.05.06 08:48:43.575164 [ 32312 ] {} <Debug> AsyncLoader: Prioritize load job 'load table shard_0.from_0': BackgroundLoad -> ForegroundLoad
2026.05.06 08:48:43.575248 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:43.575141 [ 32400 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:43.575795 [ 32373 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.575841 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:43.575938 [ 32312 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): There is no part all_0_0_0 in shard_0.from_0
2026.05.06 08:48:43.575947 [ 32314 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): There is no part all_0_0_0 in shard_0.from_1
2026.05.06 08:48:43.576137 [ 32373 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.576271 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.576301 [ 32312 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): There is no part all_1_1_0 in shard_0.from_0
2026.05.06 08:48:43.576331 [ 32314 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): There is no part all_1_1_0 in shard_0.from_1
2026.05.06 08:48:43.576420 [ 32312 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Found 0 parts that could be cloned (of 2 required parts)
2026.05.06 08:48:43.576443 [ 32314 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Found 0 parts that could be cloned (of 2 required parts)
2026.05.06 08:48:43.576576 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.576782 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:43.576893 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:43.576964 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:43.577061 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:43.577221 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:43.577374 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:43.577479 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.577621 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.577637 [ 32312 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Candidate replicas: 2
2026.05.06 08:48:43.577719 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:43.577781 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:43.577928 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:43.577979 [ 32314 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Candidate replicas: 2
2026.05.06 08:48:43.578015 [ 32314 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Candidate replica: 0
2026.05.06 08:48:43.577910 [ 32312 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Candidate replica: 0
2026.05.06 08:48:43.578913 [ 32312 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Part all_0_0_0 is not found on remote replicas
2026.05.06 08:48:43.579316 [ 32314 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Part all_2_2_0 is not found on remote replicas
2026.05.06 08:48:43.581118 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:43.581365 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:43.581427 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:43.581584 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:43.581644 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:43.581706 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:43.581842 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.581843 [ 32314 ] {} <Trace> shard_1.to (ReplicatedMergeTreeQueue): Waiting for queue-0000000000 producing  to finish before executing queue-0000000001 producing not disjoint part  (actual part all_0_3_1)
2026.05.06 08:48:43.581921 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.582003 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.582095 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.582131 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:43.581654 [ 32399 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.582215 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:43.582248 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:43.582243 [ 32312 ] {} <Information> shard_1.to (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000000 of type REPLACE_RANGE for part  (actual part all_0_3_1)because another log entry queue-0000000001 of type REPLACE_RANGE for the same part () is being processed.
2026.05.06 08:48:43.582328 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:43.582381 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:43.582398 [ 32399 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.582513 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:43.582500 [ 32312 ] {} <Information> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Will not fetch part all_0_3_1 covering all_0_0_0. Not executing log entry queue-0000000000 of type REPLACE_RANGE for part  (actual part all_0_3_1)because another log entry queue-0000000001 of type REPLACE_RANGE for the same part () is being processed.
2026.05.06 08:48:43.582741 [ 32312 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Parts covering all_0_0_0 are not found on remote replicas
2026.05.06 08:48:43.583142 [ 32312 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Candidate replicas: 2
2026.05.06 08:48:43.583180 [ 32312 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Candidate replica: 0
2026.05.06 08:48:43.583436 [ 32312 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Part all_1_1_0 is not found on remote replicas
2026.05.06 08:48:43.583476 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 3 rows, containing 3 columns (3 merged, 0 gathered) in 0.016248551 sec., 184.63184809525478 rows/sec., 2.88 KiB/sec.
2026.05.06 08:48:43.583897 [ 32400 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.584041 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Trace> MergedBlockOutputStream: filled checksums all_0_2_1 (state Temporary)
2026.05.06 08:48:43.584251 [ 32312 ] {} <Information> shard_1.to (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000000 of type REPLACE_RANGE for part  (actual part all_0_3_1)because another log entry queue-0000000001 of type REPLACE_RANGE for the same part () is being processed.
2026.05.06 08:48:43.584364 [ 32312 ] {} <Information> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Will not fetch part all_0_3_1 covering all_1_1_0. Not executing log entry queue-0000000000 of type REPLACE_RANGE for part  (actual part all_0_3_1)because another log entry queue-0000000001 of type REPLACE_RANGE for the same part () is being processed.
2026.05.06 08:48:43.584433 [ 32312 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Parts covering all_1_1_0 are not found on remote replicas
2026.05.06 08:48:43.584761 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Renaming temporary part tmp_merge_all_0_2_1 to all_0_2_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.584917 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Test> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): preparePartForCommit: inserting all_0_2_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.584974 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1) (MergerMutator): Merged 3 parts: [all_0_0_0, all_2_2_0] -> all_0_2_1
2026.05.06 08:48:43.585863 [ 32400 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.587165 [ 32312 ] {} <Trace> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Candidate replicas: 2
2026.05.06 08:48:43.587893 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Committing part all_0_2_1 to zookeeper
2026.05.06 08:48:43.588008 [ 32312 ] {} <Trace> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Candidate replica: 1
2026.05.06 08:48:43.589076 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:43.589075 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:43.589390 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:43.589390 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:43.589702 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:43.590045 [ 32312 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Will check part all_0_0_0 required for queue-0000000000, because no replicas have it (including replicas of source table)
2026.05.06 08:48:43.590211 [ 32336 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/insert_dedup_token/replicas/r1/parts/all_0_2_1
2026.05.06 08:48:43.590352 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.590359 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 312
2026.05.06 08:48:43.590458 [ 32239 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/test_1/insert_dedup_token/replicas/r1/parts/all_0_2_1
2026.05.06 08:48:43.590425 [ 32312 ] {} <Trace> shard_1.to (ReplicatedMergeTreePartCheckThread): Enqueueing all_0_0_0 for check after 0s
2026.05.06 08:48:43.590642 [ 32338 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:43.590747 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.590852 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.591025 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:43.591178 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:43.591333 [ 32005 ] {} <Information> shard_1.to (ReplicatedMergeTreePartCheckThread): Checking part all_0_0_0
2026.05.06 08:48:43.591470 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Part all_0_2_1 committed to zookeeper
2026.05.06 08:48:43.591647 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1), part: all_0_0_0
2026.05.06 08:48:43.589702 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:43.591719 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1), part: all_1_1_0
2026.05.06 08:48:43.591770 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1), part: all_2_2_0
2026.05.06 08:48:43.591829 [ 32312 ] {} <Information> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Code: 234. DB::Exception: Not found part all_0_0_0 (or part covering it) neither source table neither remote replicas. (NO_REPLICA_HAS_PART)
2026.05.06 08:48:43.591904 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.591953 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Removing successful entry queue-0000000003 from queue with type MERGE_PARTS with virtual parts [all_0_2_1]
2026.05.06 08:48:43.592042 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Adding parts [all_0_2_1] to current parts
2026.05.06 08:48:43.592117 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:43.592130 [ 32279 ] {0d2b666f-a2a0-48ee-95c0-9c76f867f0f1::all_0_2_1} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Removing part all_0_2_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.592274 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.592292 [ 32314 ] {} <Trace> shard_1.to (ReplicatedMergeTreeQueue): Waiting for queue-0000000000 producing  to finish before executing queue-0000000001 producing not disjoint part  (actual part all_0_3_1)
2026.05.06 08:48:43.592329 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 87
2026.05.06 08:48:43.592460 [ 32314 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Found part all_0_3_1 covering all_2_2_0 on replica 0
2026.05.06 08:48:43.592603 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.592691 [ 32005 ] {} <Trace> shard_1.to (ReplicatedMergeTreePartCheckThread): Part all_0_0_0 in zookeeper: false, locally: false
2026.05.06 08:48:43.593134 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:43.593193 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:43.593204 [ 32314 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Candidate replicas: 2
2026.05.06 08:48:43.593251 [ 32314 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Candidate replica: 0
2026.05.06 08:48:43.593151 [ 32005 ] {} <Information> shard_1.to (ReplicatedMergeTreePartCheckThread): Checking if anyone has a part all_0_0_0 or covering part.
2026.05.06 08:48:43.593306 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:43.593358 [ 32279 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 4.15 MiB.
2026.05.06 08:48:43.593585 [ 32006 ] {} <Test> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:48:43.593612 [ 32314 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Part all_3_3_0 is not found on remote replicas
2026.05.06 08:48:43.593665 [ 32006 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:48:43.594328 [ 32375 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:43.594651 [ 32005 ] {} <Information> shard_1.to (ReplicatedMergeTreePartCheckThread): Found part all_0_3_1 on 0 that covers the missing part all_0_0_0
2026.05.06 08:48:43.595118 [ 32006 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:48:43.595539 [ 32360 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:43.595818 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:43.595819 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:43.595940 [ 32314 ] {} <Information> shard_1.to (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000001 of type REPLACE_RANGE for part  (actual part all_0_3_1)because another log entry queue-0000000001 of type REPLACE_RANGE for the same part () is being processed.
2026.05.06 08:48:43.596014 [ 32314 ] {} <Information> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Will not fetch part all_0_3_1 covering all_3_3_0. Not executing log entry queue-0000000001 of type REPLACE_RANGE for part  (actual part all_0_3_1)because another log entry queue-0000000001 of type REPLACE_RANGE for the same part () is being processed.
2026.05.06 08:48:43.596062 [ 32314 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Parts covering all_3_3_0 are not found on remote replicas
2026.05.06 08:48:43.596126 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:43.596126 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:43.596322 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:43.596323 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:43.596561 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.596561 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.596644 [ 32347 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:43.596826 [ 32314 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:43.597034 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:43.597034 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:43.597235 [ 32314 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (Fetcher): Checking disk default with type local
2026.05.06 08:48:43.597280 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.597525 [ 32338 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.597727 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:43.597894 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:43.597952 [ 32268 ] {} <Test> Poco: Queue size: 0, current threads: 2, threads in pool: 4, current connections: 2
2026.05.06 08:48:43.598056 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:43.598111 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:43.598304 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:43.598385 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:43.598816 [ 32373 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:43.599523 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:43.599542 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:43.603006 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.604138 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.604464 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:43.605192 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.605523 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.605573 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:43.605821 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:43.606530 [ 32399 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:43.608004 [ 32375 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_176_176_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.608918 [ 32375 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): preparePartForCommit: inserting 202605_176_176_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.609721 [ 31669 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./src/Common/threadPoolCallbackRunner.h:0: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000030516e9d
7. ./build_docker/./src/Storages/StorageBuffer.cpp:0: DB::StorageBuffer::flushAllBuffers(bool) @ 0x0000000038f161ac
8. ./build_docker/./src/Storages/StorageBuffer.cpp:1096: DB::StorageBuffer::backgroundFlush() @ 0x0000000038f23398
9. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x0000000038f2757f
10. ./src/Common/Stopwatch.h:72: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000031acbc92
11. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000031ad6fb3
12. ./build_docker/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000031ad8c4f
13. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
14. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
15. ? @ 0x00007f6e2fbe4ac3
16. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:43.609997 [ 32347 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_169_169_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.610286 [ 31669 ] {} <Trace> StorageBuffer (test.hits_buffer)/Bg: Execution took 6941 ms.
2026.05.06 08:48:43.610780 [ 32360 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_176_176_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.610836 [ 32347 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): preparePartForCommit: inserting 202605_169_169_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.611046 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Fclickhouse%2Ftables%2Fto_test_uasu1ifx%2Freplicas%2F0&part=all_0_3_1&client_protocol_version=8&compress=false
2026.05.06 08:48:43.611479 [ 32542 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf) (Replicated PartsService): Sending part all_0_3_1
2026.05.06 08:48:43.611764 [ 32360 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): preparePartForCommit: inserting 202605_176_176_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.613056 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:43.613056 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:43.613347 [ 32373 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_155_155_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.616813 [ 32373 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): preparePartForCommit: inserting 202605_155_155_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.618407 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:43.622144 [ 32314 ] {} <Test> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:43.622176 [ 32385 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:43.622364 [ 32314 ] {} <Test> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:43.622790 [ 32314 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.623384 [ 32314 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.623812 [ 32314 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (Fetcher): Disk for fetch is not provided, getting disk from reservation default with type 'local'
2026.05.06 08:48:43.624397 [ 32314 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (Fetcher): Downloading part all_0_3_1 onto disk default.
2026.05.06 08:48:43.625023 [ 32360 ] {} <Trace> SystemLog (system.transactions_info_log): Flushed system log up to offset 72
2026.05.06 08:48:43.625044 [ 32314 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (Fetcher): Downloading files 9
2026.05.06 08:48:43.625167 [ 32347 ] {} <Trace> SystemLog (system.error_log): Flushed system log up to offset 3
2026.05.06 08:48:43.625925 [ 32375 ] {} <Trace> SystemLog (system.trace_log): Flushed system log up to offset 78
2026.05.06 08:48:43.626201 [ 32400 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:43.626847 [ 32373 ] {} <Trace> SystemLog (system.processors_profile_log): Flushed system log up to offset 34
2026.05.06 08:48:43.629358 [ 32399 ] {} <Trace> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_11_11_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.629705 [ 32338 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:43.629933 [ 32399 ] {} <Test> system.blob_storage_log (41777648-2291-416e-ae61-a9f74823780e): preparePartForCommit: inserting 202605_11_11_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.633068 [ 32385 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.633362 [ 32385 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.634876 [ 32314 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (Fetcher): Download of part all_0_3_1 onto disk default finished.
2026.05.06 08:48:43.635508 [ 32399 ] {} <Trace> SystemLog (system.blob_storage_log): Flushed system log up to offset 4
2026.05.06 08:48:43.637134 [ 32400 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_186_186_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.637266 [ 32314 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Renaming temporary part tmp_replace_from_fetch_all_0_3_1 to all_0_3_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.637637 [ 32400 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): preparePartForCommit: inserting 202605_186_186_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.637793 [ 32314 ] {} <Test> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): preparePartForCommit: inserting all_0_3_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.638075 [ 32314 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Trying to create zero-copy lock for part all_0_3_1
2026.05.06 08:48:43.638587 [ 32336 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/to_test_uasu1ifx/replicas/1/parts/all_0_3_1
2026.05.06 08:48:43.638845 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 226
2026.05.06 08:48:43.638945 [ 32239 ] {} <Trace> virtual void Coordination::ZooKeeperCreateRequest::writeImpl(WriteBuffer &) const: Creating part at path /clickhouse/tables/to_test_uasu1ifx/replicas/1/parts/all_0_3_1
2026.05.06 08:48:43.639254 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.641007 [ 32314 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Removing successful entry queue-0000000001 from queue with type REPLACE_RANGE with virtual parts [all_2_2_0, all_3_3_0]
2026.05.06 08:48:43.641174 [ 32314 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Entry queue-0000000001 has actual new part name all_0_3_1, removing it from mutations
2026.05.06 08:48:43.641334 [ 32314 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Removing part all_0_3_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.641540 [ 32314 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Adding parts [all_2_2_0, all_3_3_0] to current parts
2026.05.06 08:48:43.641688 [ 32314 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Removing part all_2_2_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.641772 [ 32314 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Removing part all_3_3_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.641935 [ 31995 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 67ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:43.642030 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 77
2026.05.06 08:48:43.642407 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.642821 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_149_149_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.643275 [ 32338 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): preparePartForCommit: inserting 202605_149_149_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.643568 [ 32400 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushed system log up to offset 576
2026.05.06 08:48:43.643729 [ 32283 ] {} <Debug> 107d7a0f-1932-4bf8-8e25-77087043845f::all_0_3_1 (MergeFromLogEntryTask): Skipping action for part all_0_3_1 because part all_0_3_1 already exists.
2026.05.06 08:48:43.644115 [ 32283 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Removing successful entry queue-0000000002 from queue with type MERGE_PARTS with virtual parts [all_0_3_1]
2026.05.06 08:48:43.644467 [ 32283 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Adding parts [all_0_3_1] to current parts
2026.05.06 08:48:43.644911 [ 32283 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Removing part all_0_3_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.645188 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 77
2026.05.06 08:48:43.645358 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.645529 [ 32383 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:43.648529 [ 32338 ] {} <Trace> SystemLog (system.part_log): Flushed system log up to offset 7
2026.05.06 08:48:43.651283 [ 32383 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.651462 [ 32383 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.654676 [ 31641 ] {} <Test> TCPHandler: Closing connection (open: true, cancelled: false, eof: true)
2026.05.06 08:48:43.654727 [ 31641 ] {} <Debug> TCPHandler: Done processing connection.
2026.05.06 08:48:43.655369 [ 31641 ] {} <Debug> TCP-Session-1d0fe026-ad5d-425a-979a-cb87d39af877: 1d0fe026-ad5d-425a-979a-cb87d39af877 Logout, user_id: 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:43.665796 [ 32385 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:43.674322 [ 32383 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:43.675507 [ 32385 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_156_156_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.675792 [ 32385 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): preparePartForCommit: inserting 202605_156_156_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.678611 [ 32349 ] {} <Debug> SystemLog (system.metric_log): Will use existing table system.metric_log for MetricLog
2026.05.06 08:48:43.682599 [ 32385 ] {} <Trace> SystemLog (system.query_log): Flushed system log up to offset 4
2026.05.06 08:48:43.683539 [ 32383 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_171_171_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.685968 [ 32383 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): preparePartForCommit: inserting 202605_171_171_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.690160 [ 32383 ] {} <Trace> SystemLog (system.zookeeper_log): Flushed system log up to offset 3085
2026.05.06 08:48:43.696330 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=4, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./src/Common/threadPoolCallbackRunner.h:0: std::future<std::shared_ptr<DB::MarksInCompressedFile>> std::__function::__policy_invoker<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>::__call_impl<std::__function::__default_alloc_func<std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority), std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>>(std::__function::__policy_storage const*, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority&&) @ 0x0000000039d4c7c3
7. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x0000000039d41802
8. ./build_docker/./src/Storages/MergeTree/MergeTreeMarksLoader.cpp:73: DB::MergeTreeMarksLoader::startAsyncLoad() @ 0x0000000039d40c79
9. ./build_docker/./src/Storages/MergeTree/MergeTreeReaderCompact.cpp:0: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x0000000039dc48e8
10. ./src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:0: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x0000000039ad6f57
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x0000000039acf290
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:296: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool) @ 0x0000000039e50ea4
13. ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15007]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&) @ 0x0000000039e5b639
14. ./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298: DB::createMergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, std::shared_ptr<std::atomic<unsigned long>>, bool, bool, bool) @ 0x0000000039e578a6
15. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:1136: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::createMergedStream() @ 0x000000003976c104
16. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:0: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() @ 0x000000003975ee3d
17. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:534: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::execute() @ 0x0000000039776c40
18. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:0: DB::MergeTask::execute() @ 0x000000003978bd00
19. ./build_docker/./src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:0: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000003a4ff7fd
20. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
21. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
22. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
23. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
24. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
25. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
26. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
27. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
28. ? @ 0x00007f6e2fbe4ac3
29. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:43.696382 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./src/Common/threadPoolCallbackRunner.h:0: std::future<std::shared_ptr<DB::MarksInCompressedFile>> std::__function::__policy_invoker<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>::__call_impl<std::__function::__default_alloc_func<std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority), std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>>(std::__function::__policy_storage const*, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority&&) @ 0x0000000039d4c7c3
7. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x0000000039d41802
8. ./build_docker/./src/Storages/MergeTree/MergeTreeMarksLoader.cpp:73: DB::MergeTreeMarksLoader::startAsyncLoad() @ 0x0000000039d40c79
9. ./build_docker/./src/Storages/MergeTree/MergeTreeReaderCompact.cpp:0: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x0000000039dc48e8
10. ./src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:0: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x0000000039ad6f57
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x0000000039acf290
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:296: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool) @ 0x0000000039e50ea4
13. ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15007]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&) @ 0x0000000039e5b639
14. ./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298: DB::createMergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, std::shared_ptr<std::atomic<unsigned long>>, bool, bool, bool) @ 0x0000000039e578a6
15. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:1136: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::createMergedStream() @ 0x000000003976c104
16. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:0: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() @ 0x000000003975ee3d
17. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:534: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::execute() @ 0x0000000039776c40
18. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:0: DB::MergeTask::execute() @ 0x000000003978bd00
19. ./build_docker/./src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:0: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000003a4ff7fd
20. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
21. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
22. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
23. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
24. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
25. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
26. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
27. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
28. ? @ 0x00007f6e2fbe4ac3
29. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:43.758982 [ 32315 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Executing log entry queue-0000000000 to replace parts range all_0_0_0 with 2 parts from shard_0.from_0
2026.05.06 08:48:43.759269 [ 32315 ] {} <Information> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): All parts from REPLACE PARTITION command have been already attached
2026.05.06 08:48:43.759307 [ 32359 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:43.759699 [ 32315 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Removing successful entry queue-0000000000 from queue with type REPLACE_RANGE with virtual parts [all_0_0_0, all_1_1_0]
2026.05.06 08:48:43.759769 [ 32315 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Adding parts [all_0_0_0, all_1_1_0] to current parts
2026.05.06 08:48:43.759863 [ 32315 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Removing part all_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.759942 [ 32315 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Removing part all_1_1_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:48:43.760384 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 77
2026.05.06 08:48:43.760866 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:43.764682 [ 32359 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Trying to reserve 1.65 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.764829 [ 32359 ] {} <Trace> DiskLocal: Reserved 1.65 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.793990 [ 32359 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:43.800046 [ 32359 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_311_311_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:43.800295 [ 32359 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): preparePartForCommit: inserting 202605_311_311_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:43.804343 [ 32359 ] {} <Trace> SystemLog (system.text_log): Flushed system log up to offset 6870
2026.05.06 08:48:43.829935 [ 32481 ] {} <Test> Poco: Queue size: 0, current threads: 1, threads in pool: 5, current connections: 0
2026.05.06 08:48:43.830226 [ 31641 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:58930
2026.05.06 08:48:43.834970 [ 31641 ] {} <Debug> TCPHandler: Connected ClickHouse client version 24.8.0, revision: 54472, user: default.
2026.05.06 08:48:43.835185 [ 31641 ] {} <Debug> TCP-Session-6e471a8f-583a-47dc-b294-0ac91eded53b: Authenticating user 'default' from [::1]:58930
2026.05.06 08:48:43.835461 [ 31641 ] {} <Debug> TCP-Session-6e471a8f-583a-47dc-b294-0ac91eded53b: 6e471a8f-583a-47dc-b294-0ac91eded53b Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:43.835652 [ 31641 ] {} <Debug> TCP-Session-6e471a8f-583a-47dc-b294-0ac91eded53b: Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:43.870657 [ 31950 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 307ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:43.876950 [ 31641 ] {} <Trace> TCP-Session-6e471a8f-583a-47dc-b294-0ac91eded53b: 6e471a8f-583a-47dc-b294-0ac91eded53b Creating query context from session context, user_id: 94309d50-4f52-5250-31bd-74fecac179db, parent context user: default
2026.05.06 08:48:43.878865 [ 31641 ] {f77506c0-7aac-4cb1-a224-87bcc9abcd7f} <Debug> executeQuery: (from [::1]:58930) SELECT 'Server successfully started', 'OK', NULL, '' (stage: Complete)
2026.05.06 08:48:43.881254 [ 31641 ] {f77506c0-7aac-4cb1-a224-87bcc9abcd7f} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:43.882134 [ 31641 ] {f77506c0-7aac-4cb1-a224-87bcc9abcd7f} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:43.889643 [ 31641 ] {f77506c0-7aac-4cb1-a224-87bcc9abcd7f} <Debug> executeQuery: Read 1 rows, 1.00 B in 0.011474 sec., 87.15356458079135 rows/sec., 87.15 B/sec.
2026.05.06 08:48:43.890461 [ 31641 ] {f77506c0-7aac-4cb1-a224-87bcc9abcd7f} <Debug> TCPHandler: Processed in 0.014339227 sec.
2026.05.06 08:48:43.914220 [ 32349 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:43.965273 [ 31641 ] {} <Test> TCPHandler: Closing connection (open: true, cancelled: false, eof: true)
2026.05.06 08:48:43.965382 [ 31641 ] {} <Debug> TCPHandler: Done processing connection.
2026.05.06 08:48:43.966715 [ 31641 ] {} <Debug> TCP-Session-6e471a8f-583a-47dc-b294-0ac91eded53b: 6e471a8f-583a-47dc-b294-0ac91eded53b Logout, user_id: 94309d50-4f52-5250-31bd-74fecac179db
2026.05.06 08:48:43.981014 [ 32349 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:43.981213 [ 32349 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:43.996816 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:43.996852 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:43.997285 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:43.997391 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:43.997941 [ 32131 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:43.998260 [ 32131 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.000189 [ 32447 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 2.32 GiB, peak 2.35 GiB, free memory in arenas 0.00 B, will set to 2.34 GiB (RSS), difference: 19.02 MiB
2026.05.06 08:48:44.034774 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.035379 [ 32284 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:44.068601 [ 31634 ] {} <Trace> BaseDaemon: Received signal 15
2026.05.06 08:48:44.068902 [ 31634 ] {} <Information> Application: Received termination signal (Terminated)
2026.05.06 08:48:44.069113 [ 31633 ] {} <Debug> Application: Received termination signal.
2026.05.06 08:48:44.070077 [ 31633 ] {} <Debug> Application: Waiting for current connections to close.
2026.05.06 08:48:44.149681 [ 32349 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:44.177648 [ 31918 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:44.185608 [ 31918 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:44.191572 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:44.192203 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:44.205783 [ 32464 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:44.205971 [ 32464 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:44.206564 [ 32464 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.005811307 sec. (172.078 rows/sec., 4.03 KiB/sec.)
2026.05.06 08:48:44.206679 [ 32464 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:44.210928 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:44.211505 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:44.217162 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:44.217854 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.226152 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.227991 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.228276 [ 32286 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:44.279043 [ 32349 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_135_135_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:44.279463 [ 32349 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): preparePartForCommit: inserting 202605_135_135_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:44.293849 [ 31872 ] {} <Debug> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d) (MergerMutator): Selected 16 parts from all_1_1_0 to all_16_16_0
2026.05.06 08:48:44.294278 [ 31872 ] {} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:44.294595 [ 31872 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:44.295888 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTask::PrepareStage: Merging 16 parts: from all_1_1_0 to all_16_16_0 into Compact with storage Full
2026.05.06 08:48:44.296972 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:44.297304 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.298147 [ 32362 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.298302 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.298981 [ 32461 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.299029 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.299723 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_4_4_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.299731 [ 32393 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.300293 [ 32391 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.300368 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_5_5_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.300992 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_6_6_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.300990 [ 32384 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.301541 [ 32362 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.301706 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_7_7_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.302255 [ 32461 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.302384 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_8_8_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.302971 [ 32393 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.303132 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_9_9_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.303762 [ 32391 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.303888 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_10_10_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.304367 [ 32391 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.304569 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_11_11_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.305179 [ 32362 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.305237 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_12_12_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.305796 [ 32461 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.305940 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_13_13_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.306458 [ 32393 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.306611 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_14_14_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.307239 [ 32384 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.307304 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_15_15_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.307884 [ 32391 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.307993 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_16_16_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.308522 [ 32391 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.333121 [ 32349 ] {} <Trace> SystemLog (system.metric_log): Flushed system log up to offset 2
2026.05.06 08:48:44.340824 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 16 rows, containing 3 columns (3 merged, 0 gathered) in 0.045597077 sec., 350.89968596013296 rows/sec., 10.28 KiB/sec.
2026.05.06 08:48:44.341762 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Trace> MergedBlockOutputStream: filled checksums all_1_16_1 (state Temporary)
2026.05.06 08:48:44.343279 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): Renaming temporary part tmp_merge_all_1_16_1 to all_1_16_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:44.343657 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d): preparePartForCommit: inserting all_1_16_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:44.343806 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Trace> test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d) (MergerMutator): Merged 16 parts: [all_1_1_0, all_16_16_0] -> all_1_16_1
2026.05.06 08:48:44.344067 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_1_1_0
2026.05.06 08:48:44.344198 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_2_2_0
2026.05.06 08:48:44.344294 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_3_3_0
2026.05.06 08:48:44.344387 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_4_4_0
2026.05.06 08:48:44.344480 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_5_5_0
2026.05.06 08:48:44.344584 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_6_6_0
2026.05.06 08:48:44.344676 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_7_7_0
2026.05.06 08:48:44.344769 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_8_8_0
2026.05.06 08:48:44.344968 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_9_9_0
2026.05.06 08:48:44.345069 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_10_10_0
2026.05.06 08:48:44.345163 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_11_11_0
2026.05.06 08:48:44.345256 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_12_12_0
2026.05.06 08:48:44.345350 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_13_13_0
2026.05.06 08:48:44.345443 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_14_14_0
2026.05.06 08:48:44.345571 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_15_15_0
2026.05.06 08:48:44.345672 [ 32278 ] {fb522d70-7b1a-4b77-b561-28311217550d::all_1_16_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_dst (fb522d70-7b1a-4b77-b561-28311217550d), part: all_16_16_0
2026.05.06 08:48:44.348151 [ 32278 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 4.11 MiB.
2026.05.06 08:48:44.443810 [ 31846 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.444237 [ 31846 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.444261 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.444586 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.444629 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.444954 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.447256 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.447306 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.449135 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.449293 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.449578 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:44.449820 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:44.449942 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:44.450047 [ 31840 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.450483 [ 31840 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.450589 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.450884 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.450895 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:44.451096 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.451152 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:44.451256 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.451270 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:44.452268 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:44.453481 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.453598 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.454465 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:44.455361 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.455380 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:44.455780 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:44.455846 [ 31838 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.456228 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:44.456261 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.456539 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.456789 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.460366 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.461240 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:44.461545 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:44.461767 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:44.461899 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.463533 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.464126 [ 31831 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.464509 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.464542 [ 31831 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.464804 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.464842 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.465103 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.466730 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.466981 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.468237 [ 31842 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.468369 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:44.468376 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.468553 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:44.468818 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:44.468841 [ 31826 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.469232 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.469446 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.469717 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:44.469827 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:44.469956 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:44.471069 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.471179 [ 31842 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 7755ms, ratio: 0.025641028, points per minute: 1.9838319)
2026.05.06 08:48:44.471399 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:44.471527 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:44.472085 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:44.472195 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:44.472308 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:44.472451 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.472871 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:44.473071 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:44.474258 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:44.474913 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:44.475122 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:44.475438 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:44.475554 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:44.475751 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:44.475922 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.477014 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:44.477140 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:44.477248 [ 31842 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:44.479790 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:44.482150 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:44.482226 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:44.482343 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:44.482463 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:44.482578 [ 31842 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.482673 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:44.482791 [ 31842 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:44.485267 [ 31842 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:48:44.487188 [ 31842 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:44.487467 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:44.487741 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:44.488154 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:44.488617 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.489418 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:44.489822 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:44.494989 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:44.495141 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:44.495305 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:44.495600 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.495867 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.496036 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:44.496130 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:44.496328 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:44.497245 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:44.497368 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:44.497521 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:44.497766 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.497979 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.498131 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:44.498221 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:44.498407 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:44.499430 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:44.499573 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:44.499740 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.499986 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.500187 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.500336 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.500448 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.500710 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.501701 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:44.501822 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:44.501957 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:44.502203 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.502390 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.502553 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:44.502642 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:44.502831 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:44.503953 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:44.504083 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:44.504236 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:44.504501 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.504704 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.504857 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:44.504950 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:44.505141 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:44.506149 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:44.506273 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:44.506412 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.506673 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.506873 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.507021 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.507106 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.507292 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.508101 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:44.508223 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:44.508353 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:44.508622 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.508812 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:44.508955 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:44.509042 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:44.509227 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:44.509755 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:44.510578 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:44.510979 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:44.511067 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:44.515186 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:44.516802 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.517529 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.518031 [ 31810 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.518422 [ 31810 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.518855 [ 31810 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.518898 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.519258 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.521190 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.522990 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:44.523794 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:44.523914 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:44.524015 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:44.526742 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:44.529358 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:44.529538 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:44.534842 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:44.536866 [ 31802 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:44.537086 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:44.537162 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:44.537276 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:44.541996 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:44.542891 [ 31802 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:44.546530 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:44.546721 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:44.546725 [ 32318 ] {} <Trace> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Found 18 old parts to remove. Parts: [all_1_1_0_47, all_1_1_0_48, all_1_1_0_49, all_1_1_0_50, all_1_1_0_51, all_1_1_0_52, all_1_1_0_53, all_1_1_0_54, all_1_1_0_55, all_1_1_0_56, all_1_1_0_57, all_1_1_0_58, all_1_1_0_59, all_1_1_0_60, all_1_1_0_61, all_1_1_0_62, all_1_1_0_63, all_1_1_0_64]
2026.05.06 08:48:44.547017 [ 32318 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Removing 18 parts from filesystem (concurrently): Parts: [all_1_1_0_47, all_1_1_0_48, all_1_1_0_49, all_1_1_0_50, all_1_1_0_51, all_1_1_0_52, all_1_1_0_53, all_1_1_0_54, all_1_1_0_55, all_1_1_0_56, all_1_1_0_57, all_1_1_0_58, all_1_1_0_59, all_1_1_0_60, all_1_1_0_61, all_1_1_0_62, all_1_1_0_63, all_1_1_0_64]
2026.05.06 08:48:44.547217 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:44.547814 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.548060 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:44.548659 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:44.548888 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:44.549355 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:44.551695 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_47 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.551817 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_48 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.551913 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_49 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.551979 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_50 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552043 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_51 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552107 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_52 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552170 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_53 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552235 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_54 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552298 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_55 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552361 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_56 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552425 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_57 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552502 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_58 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552565 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_59 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552629 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_60 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552695 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_61 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552758 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_62 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552821 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_63 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.552883 [ 32318 ] {} <Test> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): removePartsFinally: removing all_1_1_0_64 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.553003 [ 32318 ] {} <Debug> test_15.mt_00763_1 (b5491bcc-2091-4466-8b4d-1b87d883b7b0): Removing 18 parts from memory: Parts: [all_1_1_0_47, all_1_1_0_48, all_1_1_0_49, all_1_1_0_50, all_1_1_0_51, all_1_1_0_52, all_1_1_0_53, all_1_1_0_54, all_1_1_0_55, all_1_1_0_56, all_1_1_0_57, all_1_1_0_58, all_1_1_0_59, all_1_1_0_60, all_1_1_0_61, all_1_1_0_62, all_1_1_0_63, all_1_1_0_64]
2026.05.06 08:48:44.555205 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:44.555397 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:44.555599 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:44.555977 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.556237 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.556433 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:44.556570 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:44.556815 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:44.557963 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:44.558122 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:44.558292 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:44.558617 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.558862 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.559062 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:44.559175 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:44.559399 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:44.560751 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:44.560915 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:44.561080 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.561382 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.561637 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.561831 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.561947 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.562167 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.563331 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:44.563462 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:44.563658 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:44.563924 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.564120 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.564275 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:44.564403 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:44.564630 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:44.565863 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:44.566019 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:44.566186 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:44.566470 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.566717 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.566900 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:44.567017 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:44.567238 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:44.568527 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:44.568662 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:44.568810 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.568953 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.569082 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.569292 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.569444 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.569548 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.569763 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.570549 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:44.570669 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:44.570822 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:44.571115 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.571332 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.571511 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:44.571645 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:44.571848 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:44.572372 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:44.573375 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:44.573849 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:44.573951 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:44.577213 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:44.578624 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.579199 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.579650 [ 31787 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.580048 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.580097 [ 31787 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.580394 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.582117 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.583734 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.584013 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:44.584232 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:44.584352 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:44.584349 [ 31784 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.584779 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.584844 [ 31784 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.585117 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.585124 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:44.585163 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.585265 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:44.585382 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:44.585387 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.586008 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:44.586666 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.586869 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.587327 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:44.588078 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:44.588115 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.588234 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.588357 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:44.588684 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:44.588721 [ 31779 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.589094 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.589115 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.589160 [ 31779 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.589402 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.589428 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.589696 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.590176 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:44.590311 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:44.590430 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:44.591024 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.591290 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.592968 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.593387 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.593781 [ 31778 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.594162 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.594183 [ 31778 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.594517 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.594541 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.594794 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.595715 [ 31783 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.595823 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:44.595967 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:44.596266 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.596534 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.597962 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.598095 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:44.598164 [ 31783 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 111ms, ratio: 0.025641028, points per minute: 138.60014)
2026.05.06 08:48:44.598373 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:44.598506 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:44.598820 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:44.598953 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:44.599000 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:44.599085 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:44.599095 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:44.599243 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:44.599797 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:44.601076 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:44.601774 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:44.601960 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:44.602182 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:44.602250 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:44.602730 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.604000 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:44.604181 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:44.604313 [ 31783 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:44.604807 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:44.604999 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:44.610121 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:44.611638 [ 31783 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.611747 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:44.611890 [ 31783 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:44.612373 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:44.612452 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:44.612596 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:44.612722 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:44.614272 [ 31783 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:48:44.617340 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:44.617577 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:44.618017 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:44.618541 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.619511 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:44.619928 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:44.624724 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:44.624869 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:44.625023 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:44.625332 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.625569 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.625732 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:44.625827 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:44.626035 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:44.626945 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:44.627071 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:44.627213 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:44.627468 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.627732 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.627891 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:44.627982 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:44.628176 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:44.629213 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:44.629337 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:44.629475 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.629801 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.630054 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.630252 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.630373 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.630515 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.631135 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:44.631216 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:44.631311 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:44.631478 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.631618 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.631707 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:44.631769 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:44.631890 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:44.632452 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:44.632532 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:44.632611 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:44.632744 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.632850 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.632930 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:44.632984 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:44.633090 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:44.633625 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:44.633695 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:44.633772 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.633899 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.634000 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.634084 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.634134 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.634232 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.634619 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:44.634680 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:44.634751 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:44.634876 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.634974 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:44.635049 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:44.635094 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:44.635189 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:44.635476 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:44.635940 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:44.636152 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:44.636197 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:44.639298 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:44.639944 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.640327 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.640806 [ 31764 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.641194 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.641310 [ 31764 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.641607 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.643316 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.645186 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:44.645979 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:44.646098 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:44.646224 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:44.648987 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:44.651252 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:44.651393 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:44.655270 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:44.658665 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:44.658770 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:44.658921 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:44.659072 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:44.661744 [ 31754 ] {} <Debug> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff) (MergerMutator): Selected 20 parts from all_1_1_0 to all_20_20_0
2026.05.06 08:48:44.662258 [ 31754 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:44.662526 [ 31754 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:44.664038 [ 32282 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTask::PrepareStage: Merging 20 parts: from all_1_1_0 to all_20_20_0 into Compact with storage Full
2026.05.06 08:48:44.664145 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:44.664325 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:44.664781 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:44.665337 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.665427 [ 32282 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:44.665791 [ 32282 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.666441 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:44.666720 [ 32461 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.666866 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:44.667046 [ 32282 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.667801 [ 32393 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.668038 [ 32282 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.668726 [ 32384 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.668901 [ 32282 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_4_4_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.669627 [ 32362 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:44.669689 [ 32282 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_5_5_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:44.671514 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:44.671714 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:44.671917 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:44.672210 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.672438 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.672619 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:44.672715 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:44.672925 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:44.673055 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./src/Common/threadPoolCallbackRunner.h:0: std::future<std::shared_ptr<DB::MarksInCompressedFile>> std::__function::__policy_invoker<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>::__call_impl<std::__function::__default_alloc_func<std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority), std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>>(std::__function::__policy_storage const*, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority&&) @ 0x0000000039d4c7c3
7. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x0000000039d41802
8. ./build_docker/./src/Storages/MergeTree/MergeTreeMarksLoader.cpp:73: DB::MergeTreeMarksLoader::startAsyncLoad() @ 0x0000000039d40c79
9. ./build_docker/./src/Storages/MergeTree/MergeTreeReaderCompact.cpp:0: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x0000000039dc48e8
10. ./src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:0: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x0000000039ad6f57
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x0000000039acf290
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:296: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool) @ 0x0000000039e50ea4
13. ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15007]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&) @ 0x0000000039e5b639
14. ./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298: DB::createMergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, std::shared_ptr<std::atomic<unsigned long>>, bool, bool, bool) @ 0x0000000039e578a6
15. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:1136: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::createMergedStream() @ 0x000000003976c104
16. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:0: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() @ 0x000000003975ee3d
17. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:534: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::execute() @ 0x0000000039776c40
18. ./build_docker/./src/Storages/MergeTree/MergeTask.cpp:0: DB::MergeTask::execute() @ 0x000000003978bd00
19. ./build_docker/./src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:0: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000003a4ff7fd
20. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
21. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
22. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
23. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
24. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
25. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
26. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
27. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
28. ? @ 0x00007f6e2fbe4ac3
29. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.673860 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:44.673997 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:44.674137 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:44.674389 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.674601 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.674754 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:44.674849 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:44.675049 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:44.676170 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:44.676311 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:44.676447 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.676714 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.676920 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.677079 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.677172 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.677357 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.678334 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:44.678460 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:44.678619 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:44.678874 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.679073 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.679226 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:44.679317 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:44.679531 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:44.680650 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:44.680781 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:44.680937 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:44.681194 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.681397 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.681568 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:44.681662 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:44.681861 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:44.683070 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:44.683196 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:44.683336 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.683638 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.683839 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.684017 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.684105 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.684292 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.685062 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:44.685182 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:44.685317 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:44.685583 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.685774 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.685935 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:44.686024 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:44.686212 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:44.686741 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:44.687595 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:44.688048 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:44.688137 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:44.691692 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:44.693104 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.693851 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.694326 [ 31752 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.694756 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.694811 [ 31752 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.695128 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.696908 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.698345 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.698592 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:44.698822 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:44.698945 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:44.698936 [ 31745 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.699224 [ 31745 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.699507 [ 31745 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.699576 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.699655 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:44.699765 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:44.699888 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:44.699902 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.700475 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:44.701536 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.701934 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:44.702624 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:44.702859 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:44.703131 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.703192 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:44.703563 [ 31744 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.703730 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.704012 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.704074 [ 31744 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.704377 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.704428 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.704868 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.705213 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:44.705355 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:44.705467 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:44.706041 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.706345 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.707474 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.707870 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.708392 [ 31737 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.708895 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.708963 [ 31737 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.709240 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.709266 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.709506 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.710848 [ 31750 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.710876 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.710943 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.710953 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:44.711125 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:44.712373 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.712416 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:44.713076 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:44.713187 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:44.713299 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:44.713398 [ 31750 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 99ms, ratio: 0.025641028, points per minute: 155.40016)
2026.05.06 08:48:44.713609 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:44.713708 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:44.714224 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:44.714331 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:44.714432 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:44.714950 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:44.716145 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:44.716194 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:44.716854 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:44.717065 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:44.717352 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:44.717721 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.718444 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:44.718618 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:44.718800 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:44.718933 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:44.719042 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:44.722539 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:44.724331 [ 31750 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.724435 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:44.724597 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:44.725141 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:44.725222 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:44.725342 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:44.725460 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:44.727139 [ 31750 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:48:44.730336 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:44.730548 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:44.730954 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:44.731389 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.732591 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:44.733042 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:44.735261 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.737910 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:44.738063 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:44.738213 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:44.738509 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.738723 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.738896 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:44.739017 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:44.739244 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:44.740259 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:44.740392 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:44.740553 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:44.740810 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.741010 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.741161 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:44.741253 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:44.741442 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:44.742582 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:44.742733 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:44.742891 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.743143 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.743361 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.743537 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.743670 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.743863 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.744843 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:44.744982 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:44.745125 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:44.745388 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.745604 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.745792 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:44.745911 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:44.746133 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:44.747438 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:44.747646 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:44.747818 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:44.748144 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.748406 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.748609 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:44.748709 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:44.748907 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:44.750047 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:44.750196 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:44.750357 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.750661 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.750883 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.751065 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.751185 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.751395 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.752206 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:44.752344 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:44.752481 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:44.752752 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.752945 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:44.753090 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:44.753180 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:44.753369 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:44.753909 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:44.754729 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:44.755137 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:44.755226 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:44.758799 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:44.760362 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.761087 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.761664 [ 31719 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.762062 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.762059 [ 31719 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.762532 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.764650 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.766574 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:44.767371 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:44.767480 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:44.767595 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:44.770419 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:44.772958 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:44.773180 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:44.777259 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:44.780002 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:44.780085 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:44.780205 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:44.780324 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:44.785225 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:44.785415 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:44.785906 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:44.786318 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.787441 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:44.787901 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:44.793015 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:44.793203 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:44.793402 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:44.793707 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.793954 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.794178 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:44.794282 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:44.794518 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:44.795511 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:44.795738 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:44.795896 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:44.796207 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.796455 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.796657 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:44.796782 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:44.797009 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:44.798167 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:44.798302 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:44.798456 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.798725 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.798965 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.799152 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.799245 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.799435 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.800546 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:44.800681 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:44.800863 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:44.801197 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.801477 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.801696 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:44.801833 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:44.802072 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:44.803396 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:44.803572 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:44.803822 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:44.804148 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.804402 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.804626 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:44.804739 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:44.804996 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:44.806113 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:44.806245 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:44.806459 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.806799 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.807012 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.807167 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.807263 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.807519 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.808421 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:44.808609 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:44.808756 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:44.809029 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.809255 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:44.809448 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:44.809598 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:44.809817 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:44.810394 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:44.811316 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:44.811830 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:44.811914 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:44.815955 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:44.817180 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.817759 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.818424 [ 31714 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.818890 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.818993 [ 31714 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.819366 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.821567 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.829217 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.829450 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:44.829686 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:44.829800 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:44.829785 [ 31708 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.830203 [ 31708 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.830354 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:44.830495 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.830473 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:44.830562 [ 31708 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.830661 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:44.830751 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.831222 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:44.831775 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.832366 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:44.832683 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.832919 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:44.833055 [ 31705 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.833187 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:44.833518 [ 31705 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.833528 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:44.833810 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.833883 [ 31705 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.833984 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.834080 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.834325 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.834821 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.835112 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:44.835277 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:44.835400 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:44.835464 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.836580 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.836742 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.837116 [ 31701 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.837510 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.837913 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.838346 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.839346 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.840701 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.840784 [ 31709 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.840891 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:44.841031 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:44.841253 [ 31698 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.841700 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.841764 [ 31698 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.841991 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:44.842049 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.842198 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.843343 [ 31709 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 116ms, ratio: 0.025641028, points per minute: 132.626)
2026.05.06 08:48:44.843382 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.843562 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.843563 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:44.843700 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:44.844229 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:44.844329 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:44.844434 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:44.844759 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:44.844889 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.844982 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:44.845325 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:44.845279 [ 31693 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.845396 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:44.845464 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:44.845654 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.845912 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.846372 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:44.846833 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:44.847017 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:44.847207 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:44.847410 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.847524 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:44.847855 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:44.847936 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:44.848034 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.849118 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:44.849251 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:44.849368 [ 31709 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:44.849751 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:44.849935 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:44.850650 [ 31692 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:44.850997 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:44.851262 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:44.852428 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:44.852517 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:44.852633 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:44.852749 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:44.853284 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:44.854737 [ 31709 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.854846 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:44.854980 [ 31709 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:44.855180 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:44.856042 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:44.856156 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:44.856282 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:44.857212 [ 31709 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:48:44.857285 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:44.857463 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:44.857840 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:44.858375 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.859086 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:44.859538 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:44.860260 [ 31709 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:44.860342 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:44.863123 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:44.863295 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:44.864196 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:44.864337 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:44.864482 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:44.864796 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.865017 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.865174 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:44.865266 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:44.865466 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:44.866376 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:44.866516 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:44.866653 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:44.866900 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.867095 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.867244 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:44.867333 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:44.867533 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:44.868597 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:44.868672 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:44.868738 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:44.868892 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.869161 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.869406 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.869597 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.869704 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.869897 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.870669 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:44.870740 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:44.870851 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:44.870909 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:44.870965 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:44.871033 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:44.871173 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:44.871423 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.871666 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.871840 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:44.871931 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:44.872122 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:44.873171 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:44.873295 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:44.873435 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:44.873702 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.873903 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.874055 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:44.874158 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:44.874350 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:44.875663 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:44.875791 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:44.875870 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:44.875932 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.876057 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:44.876192 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.876388 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.876548 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.876635 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.876660 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:44.876822 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.877144 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:44.877563 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:44.877678 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:44.877808 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:44.877967 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:44.878058 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.878246 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:44.878349 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:44.878389 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:44.878478 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:44.878693 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:44.879187 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:44.879923 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:44.880320 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:44.880408 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:44.883349 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:44.883502 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:44.883633 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:44.883675 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:44.883939 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.884138 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.884287 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:44.884376 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:44.884578 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:44.885015 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.885538 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:44.885672 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:44.885687 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.885823 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:44.886124 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.886191 [ 31900 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.886333 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.886520 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:44.886611 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:44.886803 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:44.887891 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:44.888020 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:44.888157 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.888402 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.888607 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.888760 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.888849 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.889034 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.889985 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:44.890106 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:44.890238 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:44.890478 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.890701 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.890852 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:44.890942 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:44.891128 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:44.892207 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:44.892331 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:44.892470 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:44.892736 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.892935 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.893086 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:44.893178 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:44.893371 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:44.894553 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:44.894689 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:44.894827 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:44.895075 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.895281 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.895463 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:44.895571 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:44.895799 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:44.896568 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:44.896687 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:44.896822 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:44.897070 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.897261 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:44.897405 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:44.897507 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:44.897695 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:44.898219 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:44.899040 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:44.899461 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:44.899552 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:44.902417 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:44.903633 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.904223 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:44.904631 [ 31865 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.904881 [ 31865 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:44.986607 [ 32316 ] {} <Trace> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Found 5 old parts to remove. Parts: [201810_1_1_0, 201810_1_2_1, 201810_1_2_1_3, 201810_2_2_0, 201810_4_4_0]
2026.05.06 08:48:44.986849 [ 32316 ] {} <Debug> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Removing 5 parts from filesystem (serially): Parts: [201810_1_1_0, 201810_1_2_1, 201810_1_2_1_3, 201810_2_2_0, 201810_4_4_0]
2026.05.06 08:48:44.992505 [ 32316 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): removePartsFinally: removing 201810_1_1_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.992621 [ 32316 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): removePartsFinally: removing 201810_1_2_1 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.992694 [ 32316 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): removePartsFinally: removing 201810_1_2_1_3 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.992755 [ 32316 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): removePartsFinally: removing 201810_2_2_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.992815 [ 32316 ] {} <Test> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): removePartsFinally: removing 201810_4_4_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:44.992904 [ 32316 ] {} <Debug> test_13.mt_with_pk (df4c828b-3ae7-4c0a-addb-2b95e89826b2): Removing 5 parts from memory: Parts: [201810_1_1_0, 201810_1_2_1, 201810_1_2_1_3, 201810_2_2_0, 201810_4_4_0]
2026.05.06 08:48:45.000367 [ 32447 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 2.34 GiB, peak 2.35 GiB, free memory in arenas 0.00 B, will set to 2.35 GiB (RSS), difference: 8.01 MiB
2026.05.06 08:48:45.042712 [ 32316 ] {} <Trace> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Found 5 old parts to remove. Parts: [all_1_1_0, all_1_2_2, all_2_2_0, all_2_2_1, all_3_3_0]
2026.05.06 08:48:45.042919 [ 32316 ] {} <Debug> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Removing 5 parts from filesystem (serially): Parts: [all_1_1_0, all_1_2_2, all_2_2_0, all_2_2_1, all_3_3_0]
2026.05.06 08:48:45.050479 [ 32316 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): removePartsFinally: removing all_1_1_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:45.050630 [ 32316 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): removePartsFinally: removing all_1_2_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:45.050700 [ 32316 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): removePartsFinally: removing all_2_2_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:45.050763 [ 32316 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): removePartsFinally: removing all_2_2_1 (state Deleting) from data_parts_indexes
2026.05.06 08:48:45.050826 [ 32316 ] {} <Test> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): removePartsFinally: removing all_3_3_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:45.050916 [ 32316 ] {} <Debug> test_1.table_with_column_ttl (7faeb214-62a6-47c6-87cc-71ec19c3d938): Removing 5 parts from memory: Parts: [all_1_1_0, all_1_2_2, all_2_2_0, all_2_2_1, all_3_3_0]
2026.05.06 08:48:45.073013 [ 31894 ] {} <Debug> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff) (MergerMutator): Selected 20 parts from all_1_1_0 to all_20_20_0
2026.05.06 08:48:45.073617 [ 31894 ] {} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:45.073893 [ 31894 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.70 GiB.
2026.05.06 08:48:45.074992 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTask::PrepareStage: Merging 20 parts: from all_1_1_0 to all_20_20_0 into Compact with storage Full
2026.05.06 08:48:45.076282 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:45.076617 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_1_1_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.077339 [ 32391 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.077458 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_2_2_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.078096 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_3_3_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.078160 [ 32461 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.078417 [ 32312 ] {} <Trace> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Found 4 old parts to remove. Parts: [all_1_1_0, all_2_2_0, all_3_3_0, all_4_4_0]
2026.05.06 08:48:45.078526 [ 32312 ] {} <Debug> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Removing 4 parts from filesystem (serially): Parts: [all_1_1_0, all_2_2_0, all_3_3_0, all_4_4_0]
2026.05.06 08:48:45.078608 [ 32393 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.078805 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_4_4_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.079394 [ 32384 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.079519 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_5_5_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.080089 [ 32384 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.080140 [ 32312 ] {} <Test> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): removePartsFinally: removing all_1_1_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:45.080195 [ 32312 ] {} <Test> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): removePartsFinally: removing all_2_2_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:45.080221 [ 32312 ] {} <Test> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): removePartsFinally: removing all_3_3_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:45.080247 [ 32312 ] {} <Test> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): removePartsFinally: removing all_4_4_0 (state Deleting) from data_parts_indexes
2026.05.06 08:48:45.080288 [ 32312 ] {} <Debug> `01053_db`.table_for_dict (7d87e310-7994-4ec6-994e-31da877dc758): Removing 4 parts from memory: Parts: [all_1_1_0, all_2_2_0, all_3_3_0, all_4_4_0]
2026.05.06 08:48:45.080293 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_6_6_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.080911 [ 32391 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.081091 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_7_7_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.081641 [ 32461 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.081838 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_8_8_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.082380 [ 32393 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.082527 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_9_9_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.083104 [ 32393 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.083292 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_10_10_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.083909 [ 32384 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.084088 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_11_11_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.084675 [ 32391 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.084837 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_12_12_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.085479 [ 32461 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.085611 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_13_13_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.086241 [ 32362 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.086447 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_14_14_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.086969 [ 32393 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.087125 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_15_15_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.087826 [ 32384 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.087931 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_16_16_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.088610 [ 32391 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.088821 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_17_17_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.089432 [ 32461 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.089661 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_18_18_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.090162 [ 32362 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.090389 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_19_19_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.091056 [ 32393 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.091211 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_20_20_0, total 1 rows starting from the beginning of the part
2026.05.06 08:48:45.091923 [ 32384 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.096542 [ 31640 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #5
2026.05.06 08:48:45.096579 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:48:45.096966 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:45.133807 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 20 rows, containing 3 columns (3 merged, 0 gathered) in 0.059294224 sec., 337.3009823014127 rows/sec., 9.88 KiB/sec.
2026.05.06 08:48:45.134827 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Trace> MergedBlockOutputStream: filled checksums all_1_20_1 (state Temporary)
2026.05.06 08:48:45.137431 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): Renaming temporary part tmp_merge_all_1_20_1 to all_1_20_1 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.137796 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff): preparePartForCommit: inserting all_1_20_1 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.137965 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Trace> test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff) (MergerMutator): Merged 20 parts: [all_1_1_0, all_20_20_0] -> all_1_20_1
2026.05.06 08:48:45.138234 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_1_1_0
2026.05.06 08:48:45.138369 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_2_2_0
2026.05.06 08:48:45.138469 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_3_3_0
2026.05.06 08:48:45.138589 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_4_4_0
2026.05.06 08:48:45.138687 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_5_5_0
2026.05.06 08:48:45.138795 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_6_6_0
2026.05.06 08:48:45.138908 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_7_7_0
2026.05.06 08:48:45.139053 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_8_8_0
2026.05.06 08:48:45.139168 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_9_9_0
2026.05.06 08:48:45.139269 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_10_10_0
2026.05.06 08:48:45.139367 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_11_11_0
2026.05.06 08:48:45.139464 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_12_12_0
2026.05.06 08:48:45.139576 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_13_13_0
2026.05.06 08:48:45.139698 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_14_14_0
2026.05.06 08:48:45.139799 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_15_15_0
2026.05.06 08:48:45.139949 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_16_16_0
2026.05.06 08:48:45.140065 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_17_17_0
2026.05.06 08:48:45.140164 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_18_18_0
2026.05.06 08:48:45.140262 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_19_19_0
2026.05.06 08:48:45.140358 [ 32285 ] {4716bbd2-8c48-4b95-9084-b1fa45fb7aff::all_1_20_1} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: test_1.table_when_b_even (4716bbd2-8c48-4b95-9084-b1fa45fb7aff), part: all_20_20_0
2026.05.06 08:48:45.142652 [ 32285 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 4.07 MiB.
2026.05.06 08:48:45.159850 [ 32083 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1289ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:45.339935 [ 32313 ] {} <Trace> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Found 5 old parts to remove. Parts: [all_1_1_0, all_2_2_0, all_3_3_0, all_4_4_0, all_5_5_0]
2026.05.06 08:48:45.340070 [ 32313 ] {} <Debug> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Removing 5 parts from filesystem (concurrently): Parts: [all_1_1_0, all_2_2_0, all_3_3_0, all_4_4_0, all_5_5_0]
2026.05.06 08:48:45.340662 [ 32313 ] {} <Debug> test_1.test_collation (eef0f088-52c1-479c-abbd-d518de6a9d3b): Failed to remove all parts, all count 5, removed 0
2026.05.06 08:48:45.416711 [ 32138 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:45.420732 [ 32138 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.70 GiB.
2026.05.06 08:48:45.429333 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:45.430359 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:45.446501 [ 32365 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:45.446593 [ 32365 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:45.446762 [ 32365 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.003560845 sec. (280.832 rows/sec., 6.58 KiB/sec.)
2026.05.06 08:48:45.446820 [ 32365 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:45.449323 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:45.449668 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:45.453105 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:45.453556 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:45.459825 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:45.461662 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:45.461970 [ 32286 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:45.545340 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk azure
2026.05.06 08:48:45.545451 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk backups
2026.05.06 08:48:45.545496 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk cached_azure
2026.05.06 08:48:45.578805 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk default
2026.05.06 08:48:45.578917 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk disk1_02961
2026.05.06 08:48:45.578950 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk disk2_02961
2026.05.06 08:48:45.578980 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk disk3_02961
2026.05.06 08:48:45.579009 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk disk4_02961
2026.05.06 08:48:45.579046 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_cache
2026.05.06 08:48:45.579076 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_cache_2
2026.05.06 08:48:45.579107 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_cache_3
2026.05.06 08:48:45.579137 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_disk
2026.05.06 08:48:45.579167 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_disk_2
2026.05.06 08:48:45.579196 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk local_disk_3
2026.05.06 08:48:45.579481 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_cache
2026.05.06 08:48:45.579751 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_cache_02933
2026.05.06 08:48:45.579990 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_cache_02944
2026.05.06 08:48:45.580222 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_cache_multi
2026.05.06 08:48:45.580454 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_cache_multi_2
2026.05.06 08:48:45.580696 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_disk
2026.05.06 08:48:45.580748 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_disk_02944
2026.05.06 08:48:45.580779 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_disk_02963
2026.05.06 08:48:45.580811 [ 32139 ] {} <Test> DatabaseCatalog: Nothing to clean up from store/ on disk s3_no_cache
2026.05.06 08:48:45.630340 [ 32192 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:45.630653 [ 32192 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:45.630726 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:45.631044 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:45.631075 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:45.631468 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:45.633372 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:45.633551 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:45.635243 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:45.635427 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:45.635594 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:45.635671 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:45.635729 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:45.636104 [ 32042 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:45.636383 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:45.636479 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:45.636516 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:45.636578 [ 32042 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:45.636613 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:45.636840 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:45.636878 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:45.637156 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:45.637214 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:45.638547 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:45.638661 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:45.638826 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:45.639207 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:45.639418 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:45.639929 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:45.640410 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:45.641288 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:45.641299 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:45.641570 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:45.641698 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:45.641806 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:45.641928 [ 32040 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:45.642314 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:45.642377 [ 32056 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:45.642698 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:45.642759 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:45.643117 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:45.644390 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:45.644871 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:45.645906 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:45.646518 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:45.646893 [ 32034 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:45.646964 [ 32026 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:45.647059 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:45.647184 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:45.647196 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:45.647255 [ 32034 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:45.647535 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:45.647555 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:45.647842 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:45.649305 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:45.649381 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:45.649392 [ 32026 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 789ms, ratio: 0.025641028, points per minute: 19.49888)
2026.05.06 08:48:45.649574 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:45.649669 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:45.650206 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:45.650307 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:45.650402 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:45.650854 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:45.650889 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:45.650860 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:45.651297 [ 32033 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:45.651729 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:45.651901 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:45.652031 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:45.652041 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:45.652167 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:45.652366 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:45.653063 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:45.653293 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:45.653575 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:45.653880 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:45.654089 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:45.655135 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:45.655393 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:45.655570 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:45.655734 [ 32026 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:45.656168 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:45.656624 [ 32027 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:45.656981 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:45.657240 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:45.657527 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:45.657673 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:45.659200 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:45.661037 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:45.662669 [ 32026 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:45.662765 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:45.663245 [ 32026 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:45.664354 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:45.665765 [ 32026 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:48:45.667155 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:45.667229 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:45.667343 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:45.667456 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:45.667935 [ 32026 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:45.672020 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:45.672199 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:45.672660 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:45.673263 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:45.674155 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:45.674530 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:45.679017 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:45.679157 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:45.679303 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:45.679592 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.679859 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.680019 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:45.680112 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:45.680311 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:45.681218 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:45.681343 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:45.681480 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:45.681743 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.681935 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.682096 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:45.682185 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:45.682371 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:45.683399 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:45.683538 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:45.683725 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:45.683978 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.684170 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.684322 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:45.684414 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:45.684615 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:45.685636 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:45.685759 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:45.685896 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:45.686142 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.686347 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.686513 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:45.686603 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:45.686788 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:45.687867 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:45.687995 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:45.688133 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:45.688380 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.688602 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.688755 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:45.688847 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:45.689036 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:45.690045 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:45.690060 [ 31633 ] {} <Information> Application: Closed all listening sockets.
2026.05.06 08:48:45.690171 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:45.690261 [ 31633 ] {} <Information> Application: Closed connections.
2026.05.06 08:48:45.690322 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:45.690407 [ 31633 ] {} <Information> Application: Stopping AsyncLoader.
2026.05.06 08:48:45.690596 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.690731 [ 31633 ] {} <Information> CgroupsMemoryUsageObserver: Stopped cgroup current memory usage observer thread
2026.05.06 08:48:45.690834 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.690992 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:45.691080 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:45.691271 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:45.692103 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:45.692225 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:45.692361 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:45.692625 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.692818 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:45.692964 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:45.693054 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:45.693256 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:45.693778 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:45.700203 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.700666 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.700667 [ 31633 ] {} <Information> Application: Shutting down storages.
2026.05.06 08:48:45.700769 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:45.700822 [ 31633 ] {} <Trace> AsynchronousInsertQueue: Shutting down the asynchronous insertion queue
2026.05.06 08:48:45.702331 [ 31633 ] {} <Trace> AsynchronousInsertQueue: Asynchronous insertion queue finished
2026.05.06 08:48:45.702470 [ 31633 ] {} <Debug> Context: Shutting down dictionaries loader
2026.05.06 08:48:45.702816 [ 31633 ] {} <Debug> Context: Shutting down UDFs loader
2026.05.06 08:48:45.703111 [ 31633 ] {} <Debug> Context: Shutting down another UDFs storage
2026.05.06 08:48:45.703240 [ 31633 ] {} <Trace> Context: Shutting down named sessions
2026.05.06 08:48:45.703422 [ 31633 ] {} <Debug> Context: Shutting down system logs
2026.05.06 08:48:45.703695 [ 32385 ] {} <Trace> SystemLog (system.query_log): Flushing system log, 4 entries to flush up to offset 8
2026.05.06 08:48:45.703750 [ 32375 ] {} <Trace> SystemLog (system.trace_log): Flushing system log, 64 entries to flush up to offset 142
2026.05.06 08:48:45.703817 [ 32400 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushing system log, 1227 entries to flush up to offset 1803
2026.05.06 08:48:45.703849 [ 32380 ] {} <Trace> SystemLog (system.session_log): Flushing system log, 8 entries to flush up to offset 8
2026.05.06 08:48:45.703865 [ 32373 ] {} <Trace> SystemLog (system.processors_profile_log): Flushing system log, 12 entries to flush up to offset 46
2026.05.06 08:48:45.703882 [ 32360 ] {} <Trace> SystemLog (system.transactions_info_log): Flushing system log, 62 entries to flush up to offset 134
2026.05.06 08:48:45.703713 [ 32338 ] {} <Trace> SystemLog (system.part_log): Flushing system log, 65 entries to flush up to offset 72
2026.05.06 08:48:45.703852 [ 32383 ] {} <Trace> SystemLog (system.zookeeper_log): Flushing system log, 1416 entries to flush up to offset 4501
2026.05.06 08:48:45.703749 [ 32349 ] {} <Trace> SystemLog (system.metric_log): Flushing system log, 3 entries to flush up to offset 5
2026.05.06 08:48:45.703752 [ 32359 ] {} <Trace> SystemLog (system.text_log): Flushing system log, 1036 entries to flush up to offset 7906
2026.05.06 08:48:45.703922 [ 31633 ] {} <Debug> SystemLogQueue (system.query_log): Requested flush up to offset 8
2026.05.06 08:48:45.703731 [ 32347 ] {} <Trace> SystemLog (system.error_log): Flushing system log, 8 entries to flush up to offset 11
2026.05.06 08:48:45.704916 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:45.706238 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:45.706968 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:45.707435 [ 32024 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:45.707772 [ 32024 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:45.708095 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:45.708111 [ 32024 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:45.708588 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:45.710857 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:45.711905 [ 32347 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:45.712409 [ 32380 ] {} <Debug> SystemLog (system.session_log): Will use existing table system.session_log for SessionLog
2026.05.06 08:48:45.712963 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:45.713796 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:45.713902 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:45.713999 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:45.714346 [ 32375 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:45.714393 [ 32373 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:45.714354 [ 32347 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:45.714730 [ 32347 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.70 GiB.
2026.05.06 08:48:45.715364 [ 32360 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:45.716859 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:45.717772 [ 32375 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:45.717976 [ 32375 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:45.718210 [ 32373 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:45.718417 [ 32373 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:45.719260 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:45.719458 [ 32360 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:45.719712 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:45.719805 [ 32360 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:45.723286 [ 32380 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:45.723735 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:45.725959 [ 32338 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:45.726963 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:45.727036 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:45.727164 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:45.727278 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:45.728150 [ 32380 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:45.728311 [ 32380 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:45.731836 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:45.731929 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:45.732070 [ 32338 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:45.732112 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:45.732611 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:45.733166 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:45.733620 [ 32400 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:45.734167 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:45.734585 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:45.734640 [ 32375 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_2_2_0 (state Temporary)
2026.05.06 08:48:45.734943 [ 32373 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_2_2_0 (state Temporary)
2026.05.06 08:48:45.735144 [ 32347 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_2_2_0 (state Temporary)
2026.05.06 08:48:45.738862 [ 32360 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_2_2_0 (state Temporary)
2026.05.06 08:48:45.739427 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:45.739670 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:45.739847 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:45.740132 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.740366 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.740545 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:45.740659 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:45.740863 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:45.741426 [ 32400 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:45.741606 [ 32400 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:45.741846 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:45.742015 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:45.742180 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:45.742483 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.742712 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.742869 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:45.742960 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:45.743154 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:45.744331 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:45.744465 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:45.744636 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:45.744899 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.745119 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.745256 [ 32347 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Renaming temporary part tmp_insert_202605_2_2_0 to 202605_170_170_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.745284 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:45.745412 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:45.745609 [ 32347 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): preparePartForCommit: inserting 202605_170_170_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.745627 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:45.746698 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:45.746840 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:45.746989 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:45.747275 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.747573 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.747793 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:45.747898 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:45.747938 [ 32375 ] {} <Trace> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): Renaming temporary part tmp_insert_202605_2_2_0 to 202605_177_177_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.748102 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:45.748288 [ 32375 ] {} <Test> system.trace_log (aa0461d4-91df-4949-8bb0-34b45a966ffa): preparePartForCommit: inserting 202605_177_177_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.749281 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:45.749417 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:45.749582 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:45.749858 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.750076 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.750246 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:45.750365 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:45.750625 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:45.751350 [ 32373 ] {} <Trace> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): Renaming temporary part tmp_insert_202605_2_2_0 to 202605_156_156_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.751554 [ 32373 ] {} <Test> system.processors_profile_log (41d85242-310c-4d42-9b59-fea8708c8383): preparePartForCommit: inserting 202605_156_156_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.751897 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:45.752030 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:45.752210 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:45.752544 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.752783 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.752994 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:45.753091 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:45.753308 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:45.754127 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:45.754257 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:45.754405 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:45.754562 [ 32375 ] {} <Trace> SystemLog (system.trace_log): Flushed system log up to offset 142
2026.05.06 08:48:45.754682 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.754890 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:48:45.755045 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:45.755073 [ 32360 ] {} <Trace> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): Renaming temporary part tmp_insert_202605_2_2_0 to 202605_177_177_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.755136 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:45.755348 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:45.755420 [ 32360 ] {} <Test> system.transactions_info_log (99771344-2c6d-47ba-b88e-32514a4fcc9f): preparePartForCommit: inserting 202605_177_177_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.755938 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:45.756660 [ 32373 ] {} <Trace> SystemLog (system.processors_profile_log): Flushed system log up to offset 46
2026.05.06 08:48:45.756899 [ 32385 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:45.758084 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.758515 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.758574 [ 32347 ] {} <Trace> SystemLog (system.error_log): Flushed system log up to offset 11
2026.05.06 08:48:45.758606 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:45.760836 [ 32017 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9094ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:45.761065 [ 32380 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_1_1_0 (state Temporary)
2026.05.06 08:48:45.761779 [ 32360 ] {} <Trace> SystemLog (system.transactions_info_log): Flushed system log up to offset 134
2026.05.06 08:48:45.761930 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:45.763519 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:45.764331 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:45.764545 [ 32383 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:45.764761 [ 32003 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:45.767023 [ 32385 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:45.767223 [ 32385 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:45.768378 [ 32383 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:45.768464 [ 32383 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:45.768851 [ 32380 ] {} <Trace> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): Renaming temporary part tmp_insert_202605_1_1_0 to 202605_23_23_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.769095 [ 32380 ] {} <Test> system.session_log (c424f16e-4de0-4eac-ae46-e189cae50b04): preparePartForCommit: inserting 202605_23_23_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.769298 [ 32338 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_2_2_0 (state Temporary)
2026.05.06 08:48:45.773391 [ 32380 ] {} <Trace> SystemLog (system.session_log): Flushed system log up to offset 8
2026.05.06 08:48:45.779250 [ 32338 ] {} <Trace> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): Renaming temporary part tmp_insert_202605_2_2_0 to 202605_150_150_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.779427 [ 32338 ] {} <Test> system.part_log (ed4e07d5-12f1-433f-b41b-c85ea7af3040): preparePartForCommit: inserting 202605_150_150_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.782215 [ 32359 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:45.783067 [ 32338 ] {} <Trace> SystemLog (system.part_log): Flushed system log up to offset 72
2026.05.06 08:48:45.784322 [ 32359 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:45.784427 [ 32359 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:45.785077 [ 32383 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_2_2_0 (state Temporary)
2026.05.06 08:48:45.787591 [ 32400 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_2_2_0 (state Temporary)
2026.05.06 08:48:45.791630 [ 32383 ] {} <Trace> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): Renaming temporary part tmp_insert_202605_2_2_0 to 202605_172_172_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.791795 [ 32383 ] {} <Test> system.zookeeper_log (f32b09c6-f6ce-4ba9-8f8a-21554928d185): preparePartForCommit: inserting 202605_172_172_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.795057 [ 32383 ] {} <Trace> SystemLog (system.zookeeper_log): Flushed system log up to offset 4501
2026.05.06 08:48:45.796294 [ 32385 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_2_2_0 (state Temporary)
2026.05.06 08:48:45.797483 [ 32400 ] {} <Trace> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): Renaming temporary part tmp_insert_202605_2_2_0 to 202605_187_187_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.797816 [ 32359 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_2_2_0 (state Temporary)
2026.05.06 08:48:45.797851 [ 32400 ] {} <Test> system.asynchronous_metric_log (ba5f8386-a6a6-42b6-bc2d-5b7408db06cf): preparePartForCommit: inserting 202605_187_187_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.803090 [ 32400 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushed system log up to offset 1803
2026.05.06 08:48:45.805968 [ 32359 ] {} <Trace> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): Renaming temporary part tmp_insert_202605_2_2_0 to 202605_312_312_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.806131 [ 32359 ] {} <Test> system.text_log (019282f6-7d19-466e-be33-b02db2b3cf9d): preparePartForCommit: inserting 202605_312_312_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.808829 [ 32385 ] {} <Trace> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): Renaming temporary part tmp_insert_202605_2_2_0 to 202605_157_157_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:45.808910 [ 32359 ] {} <Trace> SystemLog (system.text_log): Flushed system log up to offset 7906
2026.05.06 08:48:45.808982 [ 32385 ] {} <Test> system.query_log (93c3b59f-f187-422a-93f6-a6c7174732ad): preparePartForCommit: inserting 202605_157_157_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:45.814311 [ 32385 ] {} <Trace> SystemLog (system.query_log): Flushed system log up to offset 8
2026.05.06 08:48:45.814466 [ 31633 ] {} <Debug> SystemLogQueue (system.query_thread_log): Requested flush up to offset 0
2026.05.06 08:48:45.814778 [ 31633 ] {} <Debug> SystemLogQueue (system.part_log): Requested flush up to offset 72
2026.05.06 08:48:45.815011 [ 31633 ] {} <Debug> SystemLogQueue (system.trace_log): Requested flush up to offset 142
2026.05.06 08:48:45.815185 [ 31633 ] {} <Debug> SystemLogQueue (system.crash_log): Requested flush up to offset 0
2026.05.06 08:48:45.815388 [ 31633 ] {} <Debug> SystemLogQueue (system.text_log): Requested flush up to offset 7906
2026.05.06 08:48:45.815570 [ 31633 ] {} <Debug> SystemLogQueue (system.metric_log): Requested flush up to offset 5
2026.05.06 08:48:45.899638 [ 32313 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {eef0f088-52c1-479c-abbd-d518de6a9d3b::lambda}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./src/Common/threadPoolCallbackRunner.h:0: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000030516e9d
7. ./build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:0: DB::MergeTreeData::clearPartsFromFilesystemImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>> const&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>*) @ 0x0000000039881de1
8. ./contrib/llvm-project/libcxx/include/__hash_table:935: DB::MergeTreeData::clearPartsFromFilesystem(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>> const&, bool, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>*) @ 0x000000003987abb2
9. ./build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:0: DB::MergeTreeData::clearOldPartsFromFilesystem(bool) @ 0x000000003987a4bc
10. ./build_docker/./src/Storages/StorageMergeTree.cpp:1478: bool std::__function::__policy_invoker<bool ()>::__call_impl<std::__function::__default_alloc_func<DB::StorageMergeTree::scheduleDataProcessingJob(DB::BackgroundJobsAssignee&)::$_2, bool ()>>(std::__function::__policy_storage const*) @ 0x000000003a4bd25b
11. ./src/Storages/MergeTree/IExecutableTask.h:70: DB::ExecutableLambdaAdapter::executeStep() @ 0x0000000038c5f6f7
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:298: DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397de173
13. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::threadFunction() @ 0x00000000397dfe59
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397ef19f
15. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
16. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
17. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
18. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
19. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
20. ? @ 0x00007f6e2fbe4ac3
21. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:45.961299 [ 32349 ] {} <Test> InterpreterInsertQuery: Pipeline could use up to 0 thread
2026.05.06 08:48:46.015823 [ 32349 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:48:46.015986 [ 32349 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:46.170308 [ 32349 ] {} <Trace> MergedBlockOutputStream: filled checksums 202605_2_2_0 (state Temporary)
2026.05.06 08:48:46.286671 [ 32349 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Renaming temporary part tmp_insert_202605_2_2_0 to 202605_136_136_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:46.286942 [ 32349 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): preparePartForCommit: inserting 202605_136_136_0 (state PreActive) into data_parts_indexes
2026.05.06 08:48:46.288292 [ 31901 ] {} <Debug> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a) (MergerMutator): Selected 6 parts from 202605_1_131_26 to 202605_136_136_0
2026.05.06 08:48:46.288700 [ 31901 ] {} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Trying to reserve 1.47 MiB using storage policy from min volume index 0
2026.05.06 08:48:46.288964 [ 31901 ] {} <Trace> DiskLocal: Reserved 1.47 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:46.290114 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Debug> MergeTask::PrepareStage: Merging 6 parts: from 202605_1_131_26 to 202605_136_136_0 into Compact with storage Full
2026.05.06 08:48:46.332703 [ 32349 ] {} <Trace> SystemLog (system.metric_log): Flushed system log up to offset 5
2026.05.06 08:48:46.332808 [ 31633 ] {} <Debug> SystemLogQueue (system.error_log): Requested flush up to offset 11
2026.05.06 08:48:46.333033 [ 31633 ] {} <Debug> SystemLogQueue (system.filesystem_cache_log): Requested flush up to offset 0
2026.05.06 08:48:46.333196 [ 31633 ] {} <Debug> SystemLogQueue (system.s3queue_log): Requested flush up to offset 0
2026.05.06 08:48:46.333362 [ 31633 ] {} <Debug> SystemLogQueue (system.asynchronous_metric_log): Requested flush up to offset 1803
2026.05.06 08:48:46.333661 [ 31633 ] {} <Debug> SystemLogQueue (system.opentelemetry_span_log): Requested flush up to offset 0
2026.05.06 08:48:46.333826 [ 31633 ] {} <Debug> SystemLogQueue (system.query_views_log): Requested flush up to offset 0
2026.05.06 08:48:46.334015 [ 31633 ] {} <Debug> SystemLogQueue (system.zookeeper_log): Requested flush up to offset 4501
2026.05.06 08:48:46.334176 [ 31633 ] {} <Debug> SystemLogQueue (system.session_log): Requested flush up to offset 8
2026.05.06 08:48:46.334335 [ 31633 ] {} <Debug> SystemLogQueue (system.transactions_info_log): Requested flush up to offset 134
2026.05.06 08:48:46.334506 [ 31633 ] {} <Debug> SystemLogQueue (system.processors_profile_log): Requested flush up to offset 46
2026.05.06 08:48:46.334672 [ 31633 ] {} <Debug> SystemLogQueue (system.asynchronous_insert_log): Requested flush up to offset 0
2026.05.06 08:48:46.334864 [ 31633 ] {} <Debug> SystemLogQueue (system.backup_log): Requested flush up to offset 0
2026.05.06 08:48:46.335062 [ 31633 ] {} <Debug> SystemLogQueue (system.blob_storage_log): Requested flush up to offset 4
2026.05.06 08:48:46.335256 [ 32385 ] {} <Trace> SystemLog (system.query_log): Terminating
2026.05.06 08:48:46.336108 [ 32392 ] {} <Trace> SystemLog (system.query_thread_log): Terminating
2026.05.06 08:48:46.336784 [ 32338 ] {} <Trace> SystemLog (system.part_log): Terminating
2026.05.06 08:48:46.337574 [ 32375 ] {} <Trace> SystemLog (system.trace_log): Terminating
2026.05.06 08:48:46.338468 [ 32372 ] {} <Trace> SystemLog (system.crash_log): Terminating
2026.05.06 08:48:46.339463 [ 32359 ] {} <Trace> SystemLog (system.text_log): Terminating
2026.05.06 08:48:46.410119 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:48:46.422268 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_1_131_26, total 1263 rows starting from the beginning of the part
2026.05.06 08:48:46.452247 [ 32461 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:46.477233 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_132_132_0, total 10 rows starting from the beginning of the part
2026.05.06 08:48:46.507999 [ 32362 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:46.519209 [ 32093 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:46.525810 [ 32093 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:46.533106 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_133_133_0, total 10 rows starting from the beginning of the part
2026.05.06 08:48:46.534860 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:46.535957 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:46.558593 [ 32477 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:46.558746 [ 32477 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:46.559049 [ 32477 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006736009 sec. (148.456 rows/sec., 3.48 KiB/sec.)
2026.05.06 08:48:46.559185 [ 32477 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:46.563971 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:46.564539 [ 32393 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:46.564701 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:46.571782 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:46.574315 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:46.592526 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_134_134_0, total 11 rows starting from the beginning of the part
2026.05.06 08:48:46.624835 [ 32384 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:46.651820 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_135_135_0, total 2 rows starting from the beginning of the part
2026.05.06 08:48:46.683210 [ 32391 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:46.707990 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_136_136_0, total 3 rows starting from the beginning of the part
2026.05.06 08:48:46.738562 [ 32461 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:46.757463 [ 31811 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:46.757961 [ 31811 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:46.757998 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:46.758302 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:46.758372 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:46.758664 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:46.760664 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:46.760742 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:46.763093 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:46.763170 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:46.763324 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:46.763524 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:46.763656 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:46.763644 [ 32349 ] {} <Trace> SystemLog (system.metric_log): Terminating
2026.05.06 08:48:46.763669 [ 31806 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:46.764146 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:46.764210 [ 31806 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:46.764453 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:46.764482 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:46.764547 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:46.764580 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:46.764700 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:46.764795 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:46.765351 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:46.766782 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:46.766773 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:46.767375 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:46.768207 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:46.768433 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:46.768741 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:46.768775 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:46.769062 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:46.769193 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:46.769174 [ 31803 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:46.769606 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:46.769673 [ 31803 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:46.770030 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:46.770047 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:46.770317 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:46.770672 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:46.770812 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:46.770930 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:46.771764 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:46.772005 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:46.774478 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:46.774625 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:46.774869 [ 31797 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:46.775208 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:46.775289 [ 31797 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:46.775559 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:46.775569 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:46.775836 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:46.776256 [ 31810 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:46.776355 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:46.776521 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:46.777404 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:46.777539 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:46.778938 [ 31810 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1111ms, ratio: 0.025641028, points per minute: 13.84754)
2026.05.06 08:48:46.778985 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:46.779122 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:46.779219 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:46.779252 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:46.779370 [ 31791 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:46.779739 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:46.779851 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:46.779960 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:46.779986 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:46.780059 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:46.780102 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:46.780241 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:46.780345 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:46.780595 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:46.781841 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:46.781979 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:46.782663 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:46.782850 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:46.783033 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:46.783168 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:46.783656 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:46.784601 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:46.784985 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:46.784997 [ 31788 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:46.785119 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:46.785235 [ 31810 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:46.785352 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:46.785378 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:46.785539 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:46.785607 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:46.787413 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:46.789623 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:46.789797 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:46.791310 [ 31810 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:46.791413 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:46.791561 [ 31810 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:46.791881 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:46.791965 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:46.792085 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:46.792206 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:46.794205 [ 31810 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:48:46.796585 [ 31810 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:46.796845 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:46.797030 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:46.797438 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:46.797996 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:46.798917 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:46.799295 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:46.804118 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:46.804261 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:46.804411 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:46.804703 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.804932 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.805091 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:46.805189 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:46.805392 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:46.806311 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:46.806435 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:46.806591 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:46.806844 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.807041 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.807193 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:46.807284 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:46.807472 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:46.808564 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:46.808691 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:46.808840 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:46.809086 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.809279 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.809434 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:46.809540 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:46.809728 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:46.810684 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:46.810806 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:46.810942 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:46.811186 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.811379 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.811560 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:46.811682 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:46.811871 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:46.812955 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:46.813081 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:46.813224 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:46.813471 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.813688 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.813843 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:46.813937 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:46.814129 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:46.815075 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:46.815194 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:46.815337 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:46.815600 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.815828 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.815977 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:46.816067 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:46.816271 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:46.817016 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:46.817135 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:46.817268 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:46.817529 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.817722 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:46.817869 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:46.817959 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:46.818145 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:46.818669 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:46.819376 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:46.819851 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:46.819946 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:46.824516 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:46.825888 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:46.826551 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:46.826983 [ 31779 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:46.827345 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:46.827350 [ 31779 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:46.827748 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:46.829949 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:46.831899 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:46.832846 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:46.832995 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:46.833102 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:46.834868 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:46.836145 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:46.836251 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:46.838377 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:46.841362 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:46.841450 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:46.841605 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:46.841723 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:46.846677 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:46.846873 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:46.847369 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:46.847942 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:46.849033 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:46.849442 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:46.854332 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:46.854482 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:46.854654 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:46.854956 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.855189 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.855467 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:46.855599 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:46.855848 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:46.856875 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:46.857025 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:46.857187 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:46.857452 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.857677 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.857841 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:46.857955 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:46.858166 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:46.859293 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:46.859424 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:46.859584 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:46.859893 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.860132 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.860295 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:46.860389 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:46.860598 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:46.861655 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:46.861785 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:46.861947 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:46.862215 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.862413 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.862585 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:46.862680 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:46.862877 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:46.864055 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:46.864215 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:46.864361 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:46.864635 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.864847 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.865023 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:46.865136 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:46.865330 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:46.866352 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:46.866482 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:46.866658 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:46.866914 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.867109 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.867262 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:46.867349 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:46.867578 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:46.868398 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:46.868562 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:46.868711 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:46.869000 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.869223 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:46.869376 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:46.869510 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:46.869738 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:46.870269 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:46.871118 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:46.871568 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:46.871713 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:46.879185 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:46.880575 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:46.881208 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:46.881645 [ 31771 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:46.881887 [ 31771 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:46.899705 [ 31768 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1740ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:46.966152 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./src/Common/threadPoolCallbackRunner.h:0: std::future<std::shared_ptr<DB::MarksInCompressedFile>> std::__function::__policy_invoker<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>::__call_impl<std::__function::__default_alloc_func<std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority), std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>>(std::__function::__policy_storage const*, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority&&) @ 0x0000000039d4c7c3
7. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x0000000039d41802
8. ./build_docker/./src/Storages/MergeTree/MergeTreeMarksLoader.cpp:73: DB::MergeTreeMarksLoader::startAsyncLoad() @ 0x0000000039d40c79
9. ./build_docker/./src/Storages/MergeTree/MergeTreeReaderCompact.cpp:0: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x0000000039dc48e8
10. ./src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:0: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x0000000039ad6f57
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x0000000039acf290
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:296: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool) @ 0x0000000039e50ea4
13. ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15007]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&) @ 0x0000000039e5b639
14. ./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298: DB::createMergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, std::shared_ptr<std::atomic<unsigned long>>, bool, bool, bool) @ 0x0000000039e578a6
15. ./contrib/llvm-project/libcxx/include/optional:260: DB::ReadFromPart::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000039e5f6a2
16. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:0: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000003bf82e23
17. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:296: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000003bff1445
18. ./build_docker/./src/Interpreters/MutationsInterpreter.cpp:1312: DB::MutationsInterpreter::addStreamsForLaterStages(std::vector<DB::MutationsInterpreter::Stage, std::allocator<DB::MutationsInterpreter::Stage>> const&, DB::QueryPlan&) const @ 0x00000000350d55d1
19. ./build_docker/./src/Interpreters/MutationsInterpreter.cpp:0: DB::MutationsInterpreter::execute() @ 0x00000000350dff6b
20. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::prepare() @ 0x0000000039fc79b2
21. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:2054: DB::MutateTask::execute() @ 0x0000000039fc1b76
22. ./build_docker/./src/Storages/MergeTree/MutatePlainMergeTreeTask.cpp:0: DB::MutatePlainMergeTreeTask::executeStep() @ 0x000000003a5099dc
23. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
24. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
25. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
26. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
27. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
28. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
29. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
30. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
31. ? @ 0x00007f6e2fbe4ac3
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:47.375384 [ 31692 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.375858 [ 31692 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.375870 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:47.376179 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:47.376270 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.376517 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.379066 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.379136 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.381023 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.381087 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.381401 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:47.381680 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:47.381794 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:47.381898 [ 31685 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.382285 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:47.382362 [ 31685 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.382699 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:47.382713 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.382734 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:47.382815 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:47.382926 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:47.383021 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.383751 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:47.384534 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.385088 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.385522 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:47.386428 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.386389 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:47.386738 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:47.386865 [ 31900 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.387010 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.387099 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:47.387362 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:47.387449 [ 31864 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.387570 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:47.387789 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.387829 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:47.388066 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.388915 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:47.389113 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:47.389281 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:47.389852 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.390017 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.391535 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.391576 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.391989 [ 32154 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.392349 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:47.392400 [ 32154 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.392672 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.392693 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:47.392959 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.394574 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.394680 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.396284 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.396334 [ 31684 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:47.396532 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:47.396555 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.396664 [ 32130 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.396777 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:47.397154 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:47.397196 [ 32130 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.397531 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.397625 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:47.397895 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.399439 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.399979 [ 31684 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 604ms, ratio: 0.025641028, points per minute: 25.471218)
2026.05.06 08:48:47.399987 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.400183 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:47.400286 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:47.400904 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:47.401021 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:47.401123 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:47.401301 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:47.401784 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:47.401917 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.402142 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:47.402265 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:47.402314 [ 32143 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.402371 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:47.402705 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:47.402956 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.403441 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:47.404104 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:47.404312 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:47.404595 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:47.404850 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.404978 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:47.405348 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:47.405365 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:47.405451 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:47.406738 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:47.406879 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:47.407002 [ 31684 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:47.407350 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.407711 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:47.410682 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:47.410770 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:47.410916 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:47.411058 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:47.413881 [ 31684 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:47.414028 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:47.414206 [ 31684 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:47.416536 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:47.416772 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:47.417218 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:47.417528 [ 31684 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:48:47.417803 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:47.418849 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:47.419356 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:47.420165 [ 31684 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:47.424680 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:47.424847 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:47.425035 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:47.425351 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.425620 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.425809 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:47.425931 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:47.426173 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:47.427254 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:47.427405 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:47.427573 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:47.427868 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.428114 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.428277 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:47.428372 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:47.428608 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:47.429827 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:47.429962 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:47.430106 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:47.430381 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.430625 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.430806 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:47.430905 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:47.431100 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:47.432163 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:47.432293 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:47.432434 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:47.432705 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.432902 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.433055 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:47.433148 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:47.433332 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:47.434446 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:47.434602 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:47.434749 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:47.435007 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.435212 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.435409 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:47.435553 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:47.435799 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:47.436801 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:47.436929 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:47.437075 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:47.437385 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.437613 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.437768 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:47.437857 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:47.438066 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:47.438883 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:47.439008 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:47.439164 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:47.439429 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.439686 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:47.439857 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:47.439949 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:47.440142 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:47.440784 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:47.441538 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:47.441952 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:47.442069 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:47.446539 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:47.448108 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:47.448817 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:47.449274 [ 32107 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.449688 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:47.449750 [ 32107 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:47.450108 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.452209 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.453993 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:47.454928 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:47.455065 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:47.455164 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:47.458003 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:47.460283 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:47.460431 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:47.464327 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:47.466677 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:47.466750 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:47.466878 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:47.466987 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:47.471537 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:47.471753 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:47.472316 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:47.472905 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:47.473936 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:47.474300 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:47.478781 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:47.478922 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:47.479061 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:47.479321 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.479539 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.479724 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:47.479829 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:47.480023 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:47.480920 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:47.481044 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:47.481193 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:47.481441 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.481654 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.481804 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:47.481893 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:47.482078 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:47.483095 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:47.483216 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:47.483369 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:47.483658 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.483850 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.484003 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:47.484093 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:47.484276 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:47.485267 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:47.485388 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:47.485538 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:47.485837 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.486070 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.486225 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:47.486317 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:47.486525 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:47.487582 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:47.487744 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:47.487893 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:47.488146 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.488343 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.488512 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:47.488605 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:47.488811 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:47.489757 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:47.489880 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:47.490033 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:47.490280 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.490473 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.490639 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:47.490725 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:47.490912 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:47.491672 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:47.491791 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:47.491921 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:47.492164 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.492354 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:47.492527 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:47.492616 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:47.492801 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:47.493290 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:47.493952 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:47.494359 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:47.494446 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:47.503168 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:47.504498 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:47.505114 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:47.505585 [ 32127 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:47.505801 [ 32127 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:47.669210 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 1299 rows, containing 1015 columns (1015 merged, 0 gathered) in 1.379626255 sec., 941.5593500719511 rows/sec., 1.93 MiB/sec.
2026.05.06 08:48:47.676911 [ 32166 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:47.682759 [ 32166 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:47.691882 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:47.692983 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:47.698295 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Trace> MergedBlockOutputStream: filled checksums 202605_1_136_27 (state Temporary)
2026.05.06 08:48:47.715093 [ 32458 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:47.715235 [ 32458 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:47.715554 [ 32458 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006232837 sec. (160.441 rows/sec., 3.76 KiB/sec.)
2026.05.06 08:48:47.715672 [ 32458 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:47.719683 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:47.720349 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:47.727332 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:47.729342 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:47.730817 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./src/Common/threadPoolCallbackRunner.h:0: std::future<std::shared_ptr<DB::MarksInCompressedFile>> std::__function::__policy_invoker<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>::__call_impl<std::__function::__default_alloc_func<std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority), std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>>(std::__function::__policy_storage const*, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority&&) @ 0x0000000039d4c7c3
7. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x0000000039d41802
8. ./build_docker/./src/Storages/MergeTree/MergeTreeMarksLoader.cpp:73: DB::MergeTreeMarksLoader::startAsyncLoad() @ 0x0000000039d40c79
9. ./build_docker/./src/Storages/MergeTree/MergeTreeReaderCompact.cpp:0: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x0000000039dc48e8
10. ./src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:0: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x0000000039ad6f57
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x0000000039acf290
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:296: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool) @ 0x0000000039e50ea4
13. ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15007]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&) @ 0x0000000039e5b639
14. ./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298: DB::createMergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, std::shared_ptr<std::atomic<unsigned long>>, bool, bool, bool) @ 0x0000000039e578a6
15. ./contrib/llvm-project/libcxx/include/optional:260: DB::ReadFromPart::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000039e5f6a2
16. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:0: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000003bf82e23
17. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:296: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000003bff1445
18. ./build_docker/./src/Interpreters/MutationsInterpreter.cpp:1312: DB::MutationsInterpreter::addStreamsForLaterStages(std::vector<DB::MutationsInterpreter::Stage, std::allocator<DB::MutationsInterpreter::Stage>> const&, DB::QueryPlan&) const @ 0x00000000350d55d1
19. ./build_docker/./src/Interpreters/MutationsInterpreter.cpp:0: DB::MutationsInterpreter::execute() @ 0x00000000350dff6b
20. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::prepare() @ 0x0000000039fc79b2
21. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:2054: DB::MutateTask::execute() @ 0x0000000039fc1b76
22. ./build_docker/./src/Storages/MergeTree/MutatePlainMergeTreeTask.cpp:0: DB::MutatePlainMergeTreeTask::executeStep() @ 0x000000003a5099dc
23. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
24. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
25. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
26. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
27. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
28. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
29. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
30. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
31. ? @ 0x00007f6e2fbe4ac3
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:47.763703 [ 32347 ] {} <Trace> SystemLog (system.error_log): Terminating
2026.05.06 08:48:47.764020 [ 32386 ] {} <Trace> SystemLog (system.filesystem_cache_log): Terminating
2026.05.06 08:48:47.764683 [ 32369 ] {} <Trace> SystemLog (system.s3queue_log): Terminating
2026.05.06 08:48:47.765252 [ 32400 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Terminating
2026.05.06 08:48:47.765742 [ 32404 ] {} <Trace> SystemLog (system.opentelemetry_span_log): Terminating
2026.05.06 08:48:47.766162 [ 32379 ] {} <Trace> SystemLog (system.query_views_log): Terminating
2026.05.06 08:48:47.766579 [ 32383 ] {} <Trace> SystemLog (system.zookeeper_log): Terminating
2026.05.06 08:48:47.766971 [ 32380 ] {} <Trace> SystemLog (system.session_log): Terminating
2026.05.06 08:48:47.767379 [ 32360 ] {} <Trace> SystemLog (system.transactions_info_log): Terminating
2026.05.06 08:48:47.767848 [ 32373 ] {} <Trace> SystemLog (system.processors_profile_log): Terminating
2026.05.06 08:48:47.768246 [ 32367 ] {} <Trace> SystemLog (system.asynchronous_insert_log): Terminating
2026.05.06 08:48:47.768715 [ 32361 ] {} <Trace> SystemLog (system.backup_log): Terminating
2026.05.06 08:48:47.769135 [ 32399 ] {} <Trace> SystemLog (system.blob_storage_log): Terminating
2026.05.06 08:48:47.769501 [ 31633 ] {} <Trace> Context: Shutting down database catalog
2026.05.06 08:48:47.769640 [ 31633 ] {} <Trace> DatabaseCatalog: Shutting down database 01053_db
2026.05.06 08:48:47.772099 [ 31633 ] {} <Trace> DatabaseCatalog: Shutting down database 01778_db
2026.05.06 08:48:47.772788 [ 31633 ] {} <Trace> DatabaseCatalog: Shutting down database 01945.db
2026.05.06 08:48:47.773296 [ 31633 ] {} <Trace> DatabaseCatalog: Shutting down database INFORMATION_SCHEMA
2026.05.06 08:48:47.779364 [ 31633 ] {} <Trace> DatabaseCatalog: Shutting down database datasets
2026.05.06 08:48:47.779407 [ 31633 ] {} <Trace> DatabaseCatalog: Shutting down database default
2026.05.06 08:48:47.779438 [ 31633 ] {} <Trace> DatabaseCatalog: Shutting down database information_schema
2026.05.06 08:48:47.783112 [ 31633 ] {} <Trace> DatabaseCatalog: Shutting down database rdb_test_13
2026.05.06 08:48:47.783367 [ 31633 ] {} <Trace> DatabaseCatalog: Shutting down database rdb_test_13_2
2026.05.06 08:48:47.784062 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 47
2026.05.06 08:48:47.784593 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:47.785619 [ 31633 ] {} <Trace> DatabaseCatalog: Shutting down database rdb_test_13_3
2026.05.06 08:48:47.786086 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 47
2026.05.06 08:48:47.786412 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:47.787085 [ 31633 ] {} <Trace> DatabaseCatalog: Shutting down database rdb_test_13_9
2026.05.06 08:48:47.874581 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Renaming temporary part tmp_merge_202605_1_136_27 to 202605_1_136_27 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:47.874893 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): preparePartForCommit: inserting 202605_1_136_27 (state PreActive) into data_parts_indexes
2026.05.06 08:48:47.874984 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Trace> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a) (MergerMutator): Merged 6 parts: [202605_1_131_26, 202605_136_136_0] -> 202605_1_136_27
2026.05.06 08:48:47.875147 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_1_131_26
2026.05.06 08:48:47.875224 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_132_132_0
2026.05.06 08:48:47.875274 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_133_133_0
2026.05.06 08:48:47.875322 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_134_134_0
2026.05.06 08:48:47.875370 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_135_135_0
2026.05.06 08:48:47.875420 [ 32271 ] {19242c91-8ba2-49b1-a229-6507cba84d1a::202605_1_136_27} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a), part: 202605_136_136_0
2026.05.06 08:48:47.894196 [ 32271 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 34.31 MiB.
2026.05.06 08:48:47.908317 [ 32094 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.908798 [ 32094 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.908827 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:47.909094 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:47.909204 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.909371 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.911707 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.911752 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.913626 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.913672 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.913873 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:47.914009 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:47.914112 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:47.914191 [ 32169 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.914570 [ 32169 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.914628 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:47.914749 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:47.914822 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:47.914841 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:47.914885 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:47.914899 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.915100 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.915209 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:47.916581 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:47.917081 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.917168 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.917246 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:47.917341 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:47.917743 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:47.918282 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:47.919505 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:47.919584 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:47.919655 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:47.919676 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.919742 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.920102 [ 32190 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.920359 [ 32190 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.920409 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:47.920588 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:47.920688 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.920840 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.921973 [ 32138 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:47.922030 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:47.922100 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:47.922460 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.922528 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.924300 [ 32138 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 504ms, ratio: 0.025641028, points per minute: 30.525034)
2026.05.06 08:48:47.924407 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:47.924463 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:47.924458 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:47.924458 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:47.925052 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:47.925107 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:47.925156 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:47.925327 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:47.925398 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:47.925449 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:47.925576 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:47.926648 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:47.927247 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:47.927278 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:47.927363 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:47.927680 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:47.928097 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:47.928462 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:47.928573 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:47.929394 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:47.929464 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:47.929538 [ 32138 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:47.930670 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:47.931833 [ 32138 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:47.931885 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:47.931952 [ 32138 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:47.933415 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:47.933510 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:47.933652 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:47.933798 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:47.934147 [ 32138 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 10ms, ratio: 0.025641028, points per minute: 1538.4617)
2026.05.06 08:48:47.936212 [ 32138 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:47.938832 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:47.939021 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:47.939566 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:47.940157 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:47.941123 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:47.941556 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:47.946870 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:47.947048 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:47.947226 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:47.947583 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.947967 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.948189 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:47.948311 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:47.948588 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:47.949790 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:47.949958 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:47.950138 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:47.950483 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.950739 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.950908 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:47.951011 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:47.951220 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:47.952391 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:47.952547 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:47.952691 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:47.952949 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.953163 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.953320 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:47.953417 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:47.953623 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:47.954612 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:47.954739 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:47.954880 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:47.955132 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.955330 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.955503 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:47.955598 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:47.955837 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:47.956923 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:47.957053 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:47.957198 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:47.957475 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.957706 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.957864 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:47.957961 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:47.958158 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:47.959132 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:47.959257 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:47.959401 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:47.959706 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.959907 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.960058 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:47.960148 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:47.960339 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:47.961094 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:47.961328 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:47.961465 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:47.961735 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.961933 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:47.962081 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:47.962171 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:47.962361 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:47.962895 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:47.963571 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:47.964012 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:47.964105 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:47.967990 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:47.969405 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:47.970063 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:47.970480 [ 32125 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:47.970823 [ 32125 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:47.971171 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:47.971187 [ 32125 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:47.971483 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:47.973643 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:47.975417 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:47.976297 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:47.976416 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:47.976543 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:47.979913 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:47.982950 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:47.983152 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:47.988353 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:47.990442 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:47.990551 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:47.990674 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:47.990791 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:47.995556 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:47.995782 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:47.996361 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:47.996929 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:47.998022 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:47.998404 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:48.002940 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:48.003086 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:48.003236 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:48.003530 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.003781 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.003962 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:48.004060 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:48.004263 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:48.005198 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:48.005326 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:48.005468 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:48.005743 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.005943 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.006096 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:48.006187 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:48.006376 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:48.007432 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:48.007575 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:48.007747 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:48.008001 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.008212 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.008370 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:48.008466 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:48.008674 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:48.009656 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:48.009783 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:48.009923 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:48.010185 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.010379 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.010551 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:48.010642 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:48.010832 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:48.011950 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:48.012079 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:48.012224 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:48.012516 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.012722 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.012878 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:48.012972 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:48.013167 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:48.014135 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:48.014260 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:48.014404 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:48.014676 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.014876 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.015027 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:48.015116 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:48.015306 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:48.016088 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:48.016225 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:48.016361 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:48.016633 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.016830 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:48.016977 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:48.017067 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:48.017256 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:48.017791 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:48.018437 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:48.018855 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:48.018945 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:48.022918 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:48.024342 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:48.025035 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:48.025470 [ 32192 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:48.025742 [ 32192 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:48.246428 [ 32029 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:48.252444 [ 32029 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:48.257500 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:48.258074 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:48.271997 [ 32437 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:48.272137 [ 32437 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:48.272427 [ 32437 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.004721336 sec. (211.804 rows/sec., 4.96 KiB/sec.)
2026.05.06 08:48:48.272533 [ 32437 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:48.275074 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:48.275405 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:48.278643 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:48.279091 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:48.280133 [ 32020 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:48.280458 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:48.280545 [ 32020 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:48.280739 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:48.280820 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:48.281086 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:48.282309 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:48.282578 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:48.283633 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:48.283806 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:48.283946 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:48.284045 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:48.284075 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:48.284099 [ 32008 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:48.284471 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:48.284527 [ 32008 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:48.284617 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:48.284717 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:48.284737 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:48.284768 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:48.284818 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:48.284872 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:48.285383 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:48.286081 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:48.286289 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:48.286383 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:48.286809 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:48.287445 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:48.287718 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:48.287899 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:48.288017 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:48.288007 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:48.288065 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:48.288176 [ 32271 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:48.288566 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:48.288655 [ 32013 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:48.288936 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:48.288974 [ 32013 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:48.289100 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:48.289202 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:48.289517 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:48.289783 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:48.289943 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:48.290058 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:48.290527 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:48.291130 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:48.291953 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:48.292261 [ 32016 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:48.292623 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:48.292885 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:48.292922 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:48.293304 [ 32005 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:48.293747 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:48.294040 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:48.294837 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:48.296015 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:48.296309 [ 32049 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:48.296408 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:48.296559 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:48.296675 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:48.298459 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:48.301931 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:48.302053 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:48.302158 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:48.302464 [ 32049 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 366ms, ratio: 0.025641028, points per minute: 42.03447)
2026.05.06 08:48:48.302703 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:48.302805 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:48.303462 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:48.303589 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:48.303727 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:48.304247 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:48.304901 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:48.305670 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:48.306355 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:48.306553 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:48.306863 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:48.306881 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:48.307070 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:48.307291 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:48.308635 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:48.308774 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:48.308886 [ 32049 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:48.310641 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:48.314209 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:48.314301 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:48.314450 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:48.314613 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:48.315379 [ 32049 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:48.315523 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:48.315714 [ 32049 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:48.318784 [ 32049 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:48:48.320049 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:48.320235 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:48.320731 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:48.321313 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:48.322260 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:48.322651 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:48.327530 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:48.327714 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:48.327872 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:48.328157 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.328373 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.328551 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:48.328649 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:48.328858 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:48.329806 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:48.329936 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:48.330079 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:48.330348 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.330568 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.330723 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:48.330816 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:48.331009 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:48.332115 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:48.332246 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:48.332387 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:48.332655 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.332852 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.333006 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:48.333101 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:48.333288 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:48.334260 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:48.334394 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:48.334546 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:48.334797 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.334993 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.335146 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:48.335239 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:48.335426 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:48.336535 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:48.336665 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:48.336808 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:48.337060 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.337265 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.337418 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:48.337529 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:48.337722 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:48.338703 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:48.338827 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:48.338968 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:48.339219 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.339417 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.339591 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:48.339726 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:48.339918 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:48.340675 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:48.340794 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:48.340929 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:48.341177 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.341370 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:48.341534 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:48.341642 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:48.341834 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:48.342363 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:48.343057 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:48.343467 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:48.343575 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:48.347166 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:48.347830 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:48.348133 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:48.348507 [ 31999 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:48.348796 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:48.348842 [ 31999 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:48.349062 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:48.351229 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:48.353162 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:48.354076 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:48.354192 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:48.354293 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:48.355761 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:48.356807 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:48.356894 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:48.358863 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:48.361310 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:48.361384 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:48.361513 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:48.361625 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:48.366197 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:48.366395 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:48.366933 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:48.367466 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:48.368443 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:48.368832 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:48.373326 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:48.373468 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:48.373630 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:48.373901 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.374118 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.374276 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:48.374370 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:48.374585 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:48.375476 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:48.375671 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:48.375817 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:48.376070 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.376285 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.376437 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:48.376547 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:48.376737 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:48.377787 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:48.377912 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:48.378048 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:48.378295 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.378506 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.378661 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:48.378754 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:48.378943 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:48.379938 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:48.380067 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:48.380203 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:48.380460 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.380678 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.380828 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:48.380920 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:48.381106 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:48.382148 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:48.382271 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:48.382410 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:48.382675 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.382877 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.383029 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:48.383121 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:48.383312 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:48.384376 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:48.384518 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:48.384662 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:48.384968 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.385182 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.385355 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:48.385446 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:48.385654 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:48.386431 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:48.386565 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:48.386701 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:48.386948 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.387143 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:48.387287 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:48.387376 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:48.387582 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:48.388136 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:48.388897 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:48.389283 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:48.389370 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:48.393095 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:48.394448 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:48.395113 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:48.395401 [ 32193 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:48.573041 [ 31951 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:48.574812 [ 31952 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1675ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:48.576214 [ 31951 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:48.585557 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:48.586651 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:48.609055 [ 32376 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:48.609197 [ 32376 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:48.609510 [ 32376 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006463398 sec. (154.717 rows/sec., 3.63 KiB/sec.)
2026.05.06 08:48:48.609599 [ 32376 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:48.616143 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:48.616786 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:48.623518 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:48.624245 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:48.635180 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:48.638060 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:48.638590 [ 32271 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:49.037746 [ 31893 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:49.043934 [ 31893 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:49.053355 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:49.054437 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:49.078513 [ 32442 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:49.078658 [ 32442 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:49.078957 [ 32442 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.00741733 sec. (134.819 rows/sec., 3.16 KiB/sec.)
2026.05.06 08:48:49.079043 [ 32442 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:49.083287 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:49.084084 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:49.091831 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:49.092542 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:49.103901 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.106769 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.107238 [ 32286 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:49.137159 [ 32317 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:49.137473 [ 32317 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: ::1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:49 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE72C0402C926
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:49.137718 [ 32317 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:49.143972 [ 31875 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.144395 [ 31875 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.144419 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:49.144735 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.144762 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:49.145174 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.147149 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.147257 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.149159 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.149320 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.149457 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:49.149671 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:49.149796 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:49.149808 [ 31868 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.150227 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:49.150269 [ 31868 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.150624 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:49.150629 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.150865 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:49.150960 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.150972 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:49.151142 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:49.151817 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:49.152912 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.153130 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.153639 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:49.154509 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:49.154777 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:49.155123 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.155165 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:49.155304 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.155676 [ 32087 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.155728 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:49.156219 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:49.156310 [ 32087 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.156613 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:49.156643 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.156850 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.157557 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:49.157737 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:49.157865 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:49.158258 [ 32316 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:49.158874 [ 32316 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: ::1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:49 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE72C05447394
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:49.159037 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.159084 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.159333 [ 32316 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:49.161150 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.161205 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.162953 [ 31860 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.163336 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:49.163355 [ 31860 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.163642 [ 32086 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:49.163661 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.163756 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:49.163814 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:49.163899 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:49.164080 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.165571 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.166109 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.166322 [ 32086 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 848ms, ratio: 0.025641028, points per minute: 18.142237)
2026.05.06 08:48:49.166520 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:49.166619 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:49.167254 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:49.167348 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:49.167447 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:49.167512 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:49.168008 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:49.168306 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.168534 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:49.168621 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:49.168686 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:49.169575 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:49.170250 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:49.170428 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:49.170769 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:49.171155 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:49.172022 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:49.172671 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:49.172809 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:49.172917 [ 32086 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:49.174922 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:49.175114 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:49.178240 [ 32086 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:49.178341 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:49.178459 [ 32086 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:49.180948 [ 32086 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:48:49.181777 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:49.181829 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:49.181870 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:49.181939 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:49.182009 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:49.183422 [ 32086 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:49.185524 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:49.185644 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:49.186022 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:49.186438 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:49.187289 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:49.187535 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:49.189944 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:49.190029 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:49.190122 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:49.190299 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.190425 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.190537 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:49.190595 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:49.190723 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:49.191265 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:49.191345 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:49.191430 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:49.191575 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.191694 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.191782 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:49.191841 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:49.191947 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:49.192589 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:49.192658 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:49.192736 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:49.192868 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.192982 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.193084 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:49.193141 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:49.193244 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:49.193840 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:49.193923 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:49.194006 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:49.194144 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.194245 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.194333 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:49.194382 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:49.194475 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:49.195146 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:49.195229 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:49.195319 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:49.195473 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.195641 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.195739 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:49.195799 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:49.195918 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:49.196533 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:49.196611 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:49.196702 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:49.196852 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.196965 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.197052 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:49.197102 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:49.197205 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:49.197628 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:49.197691 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:49.197769 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:49.197896 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.197999 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:49.198071 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:49.198116 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:49.198216 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:49.198554 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:49.198920 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:49.199154 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:49.199210 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:49.201776 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:49.202413 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.202761 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.203088 [ 31847 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 5685ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:49.203140 [ 31845 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:49.203452 [ 31845 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.203766 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:49.203809 [ 31845 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:49.204009 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.205380 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.206654 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:49.207202 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:49.207268 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:49.207328 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:49.208608 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:49.209625 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:49.209703 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:49.211676 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:49.213064 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:49.213110 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:49.213186 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:49.213257 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:49.216529 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:49.216640 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:49.217076 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:49.217556 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:49.218585 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:49.218804 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:49.221165 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:49.221259 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:49.221354 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:49.221539 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.221671 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.221770 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:49.221838 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:49.221964 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:49.222536 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:49.222613 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:49.222689 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:49.222844 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.222957 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.223047 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:49.223101 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:49.223206 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:49.223885 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:49.223955 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:49.224033 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:49.224180 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.224289 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.224385 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:49.224440 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:49.224555 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:49.225138 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:49.225216 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:49.225296 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:49.225441 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.225559 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.225646 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:49.225699 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:49.225803 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:49.226418 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:49.226503 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:49.226591 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:49.226746 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.226866 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.226955 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:49.227010 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:49.227110 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:49.227650 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:49.227717 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:49.227797 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:49.227929 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.228046 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.228132 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:49.228184 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:49.228286 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:49.228726 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:49.228800 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:49.228880 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:49.229025 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.229133 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.229214 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:49.229271 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:49.229380 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:49.229728 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:49.230143 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:49.230387 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:49.230436 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:49.234064 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:49.234714 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.235076 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.235456 [ 31839 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:49.235755 [ 31839 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:49.333227 [ 32316 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Unloaded primary key for outdated part 202605_135_135_0
2026.05.06 08:48:49.333340 [ 32316 ] {} <Test> system.metric_log (19242c91-8ba2-49b1-a229-6507cba84d1a): Unloaded primary key for outdated part 202605_136_136_0
2026.05.06 08:48:49.584642 [ 31801 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:49.590727 [ 31801 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:49.600569 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:49.601807 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:49.627972 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=14). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.742233 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=14). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.826411 [ 31777 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1252ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:49.856339 [ 31775 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.856821 [ 31775 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.856847 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:49.857132 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:49.857248 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.857543 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.859369 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.859455 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.861057 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.861174 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.861390 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:49.861576 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:49.861679 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:49.861721 [ 31783 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.862039 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:49.862083 [ 31783 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.862327 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.862345 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:49.862385 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:49.862502 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:49.862615 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:49.862629 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.863221 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:49.864106 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.864287 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.864895 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:49.865530 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:49.865749 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:49.865806 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.866064 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:49.866117 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.866483 [ 31767 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.866552 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:49.866874 [ 31767 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.866894 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:49.867153 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.867222 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:49.867669 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.867960 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:49.868103 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:49.868218 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:49.868950 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.869330 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.870535 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.870884 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.871228 [ 31761 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.871563 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:49.871666 [ 31761 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.871893 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.871934 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:49.872256 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.873773 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.873815 [ 31770 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:49.873919 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:49.874069 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:49.874282 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.875974 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:49.876605 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.876794 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:49.876906 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:49.877008 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:49.877036 [ 31755 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.877337 [ 31770 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 694ms, ratio: 0.025641028, points per minute: 22.168036)
2026.05.06 08:48:49.877546 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:49.877589 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:49.877712 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:49.877931 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.878447 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:49.878575 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:49.878703 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:49.879242 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:49.879752 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:49.879963 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.880926 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:49.881717 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:49.881910 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:49.881996 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:49.882130 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:49.882136 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:49.882291 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:49.882772 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:49.884354 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:49.884532 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:49.884648 [ 31770 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:49.886444 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:49.889177 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:49.889256 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:49.889379 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:49.889510 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:49.890292 [ 31770 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:49.890398 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:49.890557 [ 31770 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:49.893440 [ 31770 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:48:49.894327 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:49.894570 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:49.894986 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:49.895503 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:49.896382 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:49.896458 [ 31770 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:49.896791 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:49.901412 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:49.901587 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:49.901742 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:49.902034 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.902250 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.902410 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:49.902521 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:49.902729 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:49.903712 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:49.903892 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:49.904043 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:49.904317 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.904542 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.904737 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:49.904843 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:49.905045 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:49.906128 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:49.906252 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:49.906394 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:49.906666 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.906865 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.907020 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:49.907114 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:49.907303 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:49.908326 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:49.908452 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:49.908610 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:49.908861 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.909057 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.909212 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:49.909305 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:49.909509 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:49.910714 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:49.910878 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:49.911028 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:49.911307 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.911532 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.911717 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:49.911816 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:49.912026 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:49.913003 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:49.913127 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:49.913310 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:49.913655 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.913902 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.914097 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:49.914213 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:49.914450 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:49.915422 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:49.915673 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:49.915860 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:49.916190 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.916441 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:49.916669 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:49.916766 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:49.916964 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:49.917609 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:49.918422 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:49.918891 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:49.918986 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:49.923003 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:49.924184 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.924839 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.925365 [ 31749 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:49.925844 [ 31749 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:49.925878 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:49.926202 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:49.928247 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:49.930087 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:49.930909 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:49.931020 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:49.931122 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:49.933860 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:49.936233 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:49.936396 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:49.940529 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:49.942868 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:49.942947 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:49.943066 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:49.943197 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:49.948148 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:49.948335 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:49.948929 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:49.949524 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:49.950564 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:49.950945 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:49.956042 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:49.956193 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:49.956345 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:49.956640 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.956885 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.957045 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:49.957141 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:49.957344 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:49.958377 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:49.958535 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:49.958682 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:49.958957 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.959159 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.959314 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:49.959410 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:49.959655 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:49.960758 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:49.960889 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:49.961029 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:49.961281 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.961478 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.961649 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:49.961744 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:49.961958 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:49.963012 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:49.963151 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:49.963292 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:49.963559 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.963787 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.963944 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:49.964038 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:49.964228 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:49.965395 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:49.965553 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:49.965701 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:49.965968 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.966175 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.966334 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:49.966431 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:49.966644 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:49.967894 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:49.968026 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:49.968199 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:49.968510 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.968715 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.968870 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:49.968960 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:49.969155 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:49.969918 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:49.970037 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:49.970173 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:49.970419 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.970631 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:49.970778 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:49.970883 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:49.971075 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:49.971634 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:49.972302 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:49.972742 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:49.972833 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:49.976778 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:49.978376 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.979126 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:49.979484 [ 31737 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:50.404943 [ 31689 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 4645ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:50.611857 [ 31865 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:50.615840 [ 31865 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:50.625591 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:50.626811 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:50.649671 [ 32439 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:50.649877 [ 32439 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:50.650202 [ 32439 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006740199 sec. (148.364 rows/sec., 3.48 KiB/sec.)
2026.05.06 08:48:50.650295 [ 32439 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:50.657977 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:50.658642 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:50.665573 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:50.666293 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:50.677605 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:50.680508 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:50.680979 [ 32279 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:50.929125 [ 31895 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:50.929600 [ 31895 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:50.929691 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:50.929944 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:50.930136 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:50.930286 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:50.932597 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:50.932624 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:50.934292 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:50.934334 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:50.934537 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:50.934667 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:50.934764 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:50.934864 [ 32171 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:50.935219 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:50.935377 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:50.935455 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:50.935394 [ 32152 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:50.935565 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:50.935777 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:50.935914 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:50.936025 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:50.936237 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:50.937162 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:50.937399 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:50.937935 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:50.938033 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:50.938271 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:50.938682 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:50.939150 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:50.940066 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:50.940419 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:50.940477 [ 32161 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:50.940587 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:50.940634 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:50.940739 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:50.941014 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:50.941035 [ 32161 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:50.941343 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:50.941384 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:50.941689 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:50.943055 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:50.943397 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:50.944627 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:50.944947 [ 32092 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:50.945073 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:50.945407 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:50.945456 [ 32092 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:50.945718 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:50.945730 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:50.945929 [ 32108 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:50.945993 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:50.946030 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:50.946178 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:50.947341 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:50.947433 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:50.948437 [ 32108 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1052ms, ratio: 0.025641028, points per minute: 14.62416)
2026.05.06 08:48:50.948639 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:50.948735 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:50.948896 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:50.948917 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:50.949327 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:50.949324 [ 31863 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:50.949423 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:50.949537 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:50.949716 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:50.949720 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:50.949881 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:50.949959 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:50.949989 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:50.950029 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:50.951233 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:50.951402 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:50.951929 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:50.952151 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:50.952460 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:50.952740 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:50.952968 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:50.953808 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:50.954127 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:50.954190 [ 32183 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:50.954259 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:50.954370 [ 32108 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:50.954536 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:50.954838 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:50.955055 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:50.955199 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:50.956536 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:50.958157 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:50.959542 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:50.961176 [ 32108 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:50.961282 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:50.961407 [ 32108 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:50.962170 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:50.962274 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:50.962438 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:50.962621 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:50.964028 [ 32108 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:48:50.966032 [ 32108 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:50.967857 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:50.968053 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:50.968533 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:50.969113 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:50.970073 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:50.970627 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:50.975778 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:50.975932 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:50.976087 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:50.976376 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.976611 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.976775 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:50.976873 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:50.977084 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:50.978048 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:50.978176 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:50.978317 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:50.978593 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.978855 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.979073 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:50.979200 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:50.979453 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:50.980658 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:50.980794 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:50.980938 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:50.981193 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.981394 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.981574 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:50.981685 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:50.981878 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:50.982865 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:50.982989 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:50.983129 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:50.983378 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.983590 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.983777 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:50.983872 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:50.984098 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:50.985483 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:50.985644 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:50.985798 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:50.986070 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.986281 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.986456 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:50.986589 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:50.986801 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:50.988104 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:50.988238 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:50.988388 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:50.988663 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.988889 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.989105 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:50.989230 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:50.989477 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:50.990524 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:50.990695 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:50.990873 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:50.991196 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.991465 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:48:50.991723 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:50.991848 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:50.992094 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:50.992817 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:50.993718 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:50.994284 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:50.994395 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:50.998726 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:51.000194 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:51.000878 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:51.001334 [ 32196 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:51.001731 [ 32196 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:51.002055 [ 32196 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:51.002163 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:51.002559 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:51.004749 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:51.006737 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:51.007664 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:51.007784 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:51.007892 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:51.010688 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:51.013043 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:51.013213 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:51.017286 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:51.019791 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:51.019876 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:51.020004 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:51.020124 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:51.024910 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:51.025105 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:51.025624 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:51.026241 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:51.027432 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:51.027889 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:51.032810 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:51.032962 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:51.033117 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:51.033424 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.033664 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.033829 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:51.033928 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:51.034135 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:51.035171 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:51.035342 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:51.035544 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:51.035912 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.036163 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.036363 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:51.036468 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:51.036705 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:51.037822 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:51.037956 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:51.038100 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:51.038359 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.038581 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.038738 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:51.038849 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:51.039043 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:51.040199 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:51.040342 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:51.040510 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:51.040769 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.040967 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.041124 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:51.041219 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:51.041410 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:51.042539 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:51.042671 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:51.042819 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:51.043085 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.043296 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.043467 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:51.043583 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:51.043834 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:51.044847 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:51.044975 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:51.045120 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:51.045376 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.045599 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.045752 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:51.045843 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:51.046036 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:51.046816 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:51.046939 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:51.047076 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:51.047326 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.047555 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:51.047735 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:51.047830 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:51.048026 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:51.048605 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:51.049277 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:51.049708 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:51.049802 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:51.054005 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:51.055558 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:51.056297 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:51.056884 [ 32181 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:51.057144 [ 32181 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:51.109570 [ 31928 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1283ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:51.774190 [ 32034 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:51.780273 [ 32034 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:51.789296 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:51.790360 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:51.815891 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:51.816455 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:51.973620 [ 32010 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:51.973909 [ 32010 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:51.974012 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:51.974183 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:51.974374 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:51.974553 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:51.976863 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:51.977024 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:51.978909 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:51.978982 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:51.979100 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:51.979207 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:51.979273 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:51.979330 [ 32008 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:51.979733 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:51.979742 [ 32008 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:51.980052 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:51.980096 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:51.980150 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:51.980168 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:51.980345 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:51.980431 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:51.981070 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:51.981941 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:51.982324 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:51.982817 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:51.983729 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:51.983703 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:51.983962 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:51.984072 [ 32007 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:51.984314 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:51.984400 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:51.984444 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:51.984737 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:51.984957 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:51.986285 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:51.986427 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:51.986584 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:51.987054 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:51.989245 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:51.989622 [ 32006 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:51.989980 [ 32006 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:51.989987 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:51.990256 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:51.990279 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:51.990538 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:51.992565 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:51.992599 [ 32009 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:51.992651 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:51.992703 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:51.992862 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:51.994958 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:51.995014 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:51.995370 [ 32017 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:51.995744 [ 32009 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1029ms, ratio: 0.025641028, points per minute: 14.951036)
2026.05.06 08:48:51.995782 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:51.995896 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:51.995960 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:51.996010 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:51.996059 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:51.996061 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:51.996111 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:51.996731 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:51.996832 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:51.996940 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:51.997527 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:51.998151 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:51.998914 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:51.999035 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:51.999663 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:51.999867 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:51.999895 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.000201 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:52.000295 [ 32001 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.000634 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:52.000684 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:52.000916 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.001473 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:52.001644 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:52.002142 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:52.002297 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:52.002430 [ 32009 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:52.002907 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.005049 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.005516 [ 31998 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.005950 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:52.006049 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:52.006423 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.008535 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:52.008595 [ 32009 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:52.008614 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:52.008710 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:52.008726 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.008766 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:52.008836 [ 32009 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:52.008902 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:52.010624 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:52.011212 [ 32009 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:48:52.011446 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:52.011655 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:52.011797 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:52.013618 [ 32009 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:52.013805 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:52.013990 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:52.014450 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:52.014664 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:52.015068 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:52.016004 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:52.016387 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:52.016939 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:52.017077 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:52.020979 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:52.021376 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:52.021557 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:52.021721 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:52.022031 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.022266 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.022440 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:52.022555 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:52.022764 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:52.023414 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:52.023520 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:52.023678 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:52.023799 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:52.023803 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:52.023947 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:52.024090 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:52.024355 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.024574 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.024731 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:52.024824 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:52.025019 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:52.026115 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:52.026240 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:52.026379 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.026650 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.026890 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.027084 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.027202 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.027441 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.028644 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:52.028780 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:52.028890 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:52.028924 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:52.029080 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:52.029181 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.029421 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.029625 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:52.029630 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:52.029781 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:52.029975 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:52.030226 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:52.031105 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:52.031211 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:52.031234 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:52.031378 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:52.031662 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:52.031704 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.031919 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.032076 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:52.032171 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:52.032366 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:52.033372 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:52.033513 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:52.033656 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.033910 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.034105 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.034255 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.034343 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.034549 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.035304 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:52.035423 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:52.035573 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:52.035878 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.036079 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:52.036227 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:52.036226 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:52.036329 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:52.036426 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:52.036537 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:52.036602 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:52.036895 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.037064 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:52.037115 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.037280 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:52.037395 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:52.037644 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:52.037804 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:52.038217 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:52.038309 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:52.038638 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:52.038782 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:52.038929 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:52.039194 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.039412 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.039590 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:52.039730 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:52.039927 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:52.041047 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:52.041175 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:52.041316 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.041589 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.041788 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.041902 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:52.041949 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.042047 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.042267 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.043268 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:52.043410 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:52.043397 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.043571 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:52.043889 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.044087 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.044241 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:52.044207 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.044335 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:52.044541 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:52.044659 [ 31994 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.045649 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:52.045779 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:52.045924 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:52.046184 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.046418 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.046644 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:52.046749 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:52.046954 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:52.048001 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:52.048133 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:52.048296 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.048577 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.048781 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.048935 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.049026 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.049236 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.050025 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:52.050148 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:52.050284 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:52.050573 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.050774 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.050938 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:52.051066 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:52.051285 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:52.052009 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:52.052712 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:52.053124 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:52.053215 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:52.056851 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:52.058402 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.059265 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.059697 [ 32187 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.072569 [ 31992 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 8554ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:52.262209 [ 31970 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.262470 [ 31970 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.262564 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:52.262767 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:52.262859 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.263097 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.265177 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.265267 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.267049 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.267118 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.267312 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:52.267549 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:52.267624 [ 31966 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.267734 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:52.267890 [ 31966 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.267943 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:52.268178 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:52.268225 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.268583 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.268591 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:52.268771 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:52.268883 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:52.269578 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:52.270020 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.270672 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.271255 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:52.271823 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.271994 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:52.272208 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:52.272231 [ 31964 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.272379 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.272649 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:52.272649 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:52.272796 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.273140 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:52.274388 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:52.274578 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:52.274706 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:52.274819 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.276113 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.276444 [ 31959 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.276896 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:52.276914 [ 31959 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.277161 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.277181 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:52.277352 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.278782 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.278822 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.280381 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.280408 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.280575 [ 31965 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:52.280688 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:52.280808 [ 31955 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.280846 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:52.281195 [ 31955 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.281239 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:52.281481 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:52.281735 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.281784 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.283263 [ 31965 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 270ms, ratio: 0.025641028, points per minute: 56.98006)
2026.05.06 08:48:52.283443 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:52.283476 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.283554 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:52.283573 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.284208 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:52.284302 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:52.284400 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:52.284905 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:52.285295 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:52.285348 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.286121 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:52.286234 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:52.286303 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:52.286340 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:52.286968 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:52.287145 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:52.287515 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:52.288038 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:52.289230 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:52.289336 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:52.289480 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:52.289605 [ 31965 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:52.291555 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:52.291749 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:52.294829 [ 31965 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:52.294928 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:52.295050 [ 31965 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:52.297618 [ 31965 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:48:52.298115 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:52.300110 [ 31965 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:52.300450 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:52.300545 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:52.300668 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:52.300783 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:52.305508 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:52.305693 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:52.306105 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:52.306747 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:52.307708 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:52.308073 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:52.312629 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:52.312774 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:52.312939 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:52.313222 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.313435 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.313614 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:52.313712 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:52.313918 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:52.314854 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:52.314981 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:52.315125 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:52.315394 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.315641 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.315801 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:52.315894 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:52.316088 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:52.317172 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:52.317296 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:52.317436 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.317705 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.317901 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.318057 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.318151 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.318340 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.319311 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:52.319434 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:52.319590 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:52.319884 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.320079 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.320233 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:52.320326 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:52.320539 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:52.321609 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:52.321734 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:52.321877 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:52.322129 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.322332 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.322486 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:52.322595 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:52.322789 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:52.323783 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:52.323909 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:52.324050 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.324300 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.324512 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.324675 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.324764 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.324955 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.325701 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:52.325820 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:52.325953 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:52.326195 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.326389 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:48:52.326552 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:52.326642 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:52.326832 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:52.327345 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:52.328118 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:52.328541 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:52.328632 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:52.331899 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:52.333428 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.334154 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.334524 [ 31942 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.334806 [ 31942 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.334938 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:52.335278 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.336966 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.338577 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:52.339330 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:52.339436 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:52.339554 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:52.342474 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:52.344935 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:52.345092 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:52.349522 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:52.352343 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:52.352428 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:52.352573 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:52.352690 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:52.357392 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:52.357595 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:52.358103 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:52.358660 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:52.359657 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:52.360053 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:52.364832 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:52.364983 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:52.365133 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:52.365428 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.365676 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.365839 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:52.365937 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:52.366147 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:52.367079 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:52.367207 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:52.367349 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:52.367656 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.367863 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.368019 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:52.368113 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:52.368306 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:52.369520 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:52.369667 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:52.369809 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.370061 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.370259 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.370414 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.370529 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.370718 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.371833 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:52.371967 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:52.372108 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:52.372370 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.372614 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.372773 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:52.372867 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:52.373059 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:52.374153 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:52.374278 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:52.374421 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:52.374694 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.374903 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.375057 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:52.375152 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:52.375348 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:52.376396 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:52.376548 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:52.376694 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.376949 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.377149 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.377300 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.377405 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.377614 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.378360 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:52.378478 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:52.378633 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:52.378882 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.379077 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.379225 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:52.379315 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:52.379519 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:52.380081 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:52.380785 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:52.381186 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:52.381276 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:52.385709 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:52.387012 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.387701 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.388138 [ 31931 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.388450 [ 31931 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.505075 [ 31921 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.505518 [ 31921 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.505584 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:52.505941 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.508144 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.509905 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.510154 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:52.510295 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:52.510397 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:52.510417 [ 31980 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.510723 [ 31980 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.511036 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:52.511113 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:52.511133 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:52.511132 [ 31980 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.511274 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:52.511524 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.511881 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:52.513413 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:52.513448 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.514112 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:52.514318 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:52.514769 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:52.515230 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:52.516301 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.516689 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:52.516829 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:52.516805 [ 31916 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.516941 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:52.517260 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:52.517278 [ 31916 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.517603 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.517638 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:52.517904 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.519777 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.519862 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.521799 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.521885 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.522263 [ 31909 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.522308 [ 31923 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:52.522411 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:52.522553 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:52.522603 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:52.522680 [ 31909 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.522930 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.523026 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:52.523398 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.524753 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.524925 [ 31923 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 225ms, ratio: 0.025641028, points per minute: 68.376076)
2026.05.06 08:48:52.525158 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:52.525258 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:52.525617 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.526078 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:52.526143 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:52.526203 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:52.526450 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:52.526749 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.527037 [ 31906 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.527273 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:52.527375 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:52.527441 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:52.527716 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.527974 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:52.528047 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:52.528080 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:52.528197 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:52.528316 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:52.528343 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:52.528728 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:52.529703 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:52.529786 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:52.529824 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.529848 [ 31923 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:52.530075 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:52.531352 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:52.531368 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:52.531455 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:52.532768 [ 31923 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:52.532831 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:52.532905 [ 31923 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:52.534529 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:52.534736 [ 31923 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 9ms, ratio: 0.025641028, points per minute: 1709.4017)
2026.05.06 08:48:52.536486 [ 31923 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:52.537428 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:52.537551 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:52.537702 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:52.537849 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:52.543093 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:52.543210 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:52.544108 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:52.544758 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:52.545796 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:52.546011 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:52.548051 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:52.548134 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:52.548220 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:52.548374 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.548501 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.548668 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:52.548764 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:52.548968 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:52.549904 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:52.550029 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:52.550189 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:52.550442 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.550662 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.550815 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:52.550906 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:52.551099 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:52.552212 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:52.552347 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:52.552512 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.552768 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.552965 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.553120 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.553215 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.553405 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.554393 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:52.554536 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:52.554675 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:52.554925 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.555117 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.555271 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:52.555364 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:52.555569 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:52.556667 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:52.556798 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:52.556940 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:52.557193 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.557397 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.557567 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:52.557662 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:52.557872 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:52.558838 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:52.558961 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:52.559103 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.559352 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.559565 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.559766 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.559860 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.560050 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.560807 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:52.560926 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:52.561061 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:52.561307 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.561518 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:52.561677 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:52.561768 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:52.561956 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:52.562505 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:52.563187 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:52.563600 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:52.563727 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:52.567345 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:52.568697 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.569341 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.569779 [ 31897 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.570112 [ 31897 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:52.570412 [ 31897 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.570577 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:52.570981 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:52.573106 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:52.575147 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:52.576083 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:52.576212 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:52.576320 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:52.579074 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:52.581361 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:52.581517 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:52.585517 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:52.588442 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:52.588579 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:52.588788 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:52.588967 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:52.594131 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:52.594366 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:52.594917 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:52.595486 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:52.596714 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:52.597129 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:52.602586 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:52.602758 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:52.602918 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:52.603216 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.603443 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.603762 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:52.603915 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:52.604195 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:52.605273 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:52.605459 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:52.605642 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:52.605944 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.606156 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.606323 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:52.606466 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:52.606703 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:52.607941 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:52.608084 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:52.608232 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.608594 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.608887 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.609068 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.609170 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.609395 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.610542 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:52.610684 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:52.610834 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:52.611106 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.611309 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.611540 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:52.611685 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:52.611890 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:52.613086 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:52.613219 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:52.613395 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:52.613795 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.614089 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.614299 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:52.614408 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:52.614639 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:52.615843 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:52.615982 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:52.616177 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:52.616462 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.616688 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.616845 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:52.616939 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:52.617180 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:52.618004 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:52.618186 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:52.618392 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:52.618743 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.618961 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:52.619161 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:52.619285 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:52.619507 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:52.620039 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:52.620699 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:52.621083 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:52.621183 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:52.625466 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:52.626606 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.627161 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.627630 [ 32112 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.627912 [ 32112 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:52.803910 [ 32097 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 3601ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:52.937296 [ 32081 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:52.943433 [ 32081 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:52.949435 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:52.950169 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:52.965747 [ 32432 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:52.965937 [ 32432 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:52.966336 [ 32432 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.005631318 sec. (177.578 rows/sec., 4.16 KiB/sec.)
2026.05.06 08:48:52.966454 [ 32432 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:52.969986 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:52.970407 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:52.974448 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:52.974940 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:52.982117 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.984374 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:52.984751 [ 32280 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:53.073993 [ 31855 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1965ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:53.312520 [ 31825 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:53.312978 [ 31825 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:53.313003 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:53.313337 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:53.313347 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:53.313654 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:53.315681 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:53.315800 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:53.317477 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:53.317646 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:53.317704 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:53.317807 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:53.318134 [ 31821 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:53.318479 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:53.318577 [ 31821 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:53.318632 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:53.318701 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:53.318774 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:53.318794 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:53.318887 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:53.319156 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:53.319201 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:53.320754 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:53.320790 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:53.321228 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:53.321309 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:53.321440 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:53.321846 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:53.322334 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:53.323342 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:53.323717 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:53.323802 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:53.323869 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:53.323889 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:53.324208 [ 31814 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:53.324599 [ 31814 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:53.324659 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:53.324882 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:53.325072 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:53.325143 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:53.326791 [ 31817 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:53.326855 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:53.326949 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:53.327259 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:53.327275 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:53.329038 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:53.329051 [ 31817 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 793ms, ratio: 0.025641028, points per minute: 19.400524)
2026.05.06 08:48:53.329196 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:53.330059 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:53.330192 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:53.330317 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:53.331298 [ 31817 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:53.331394 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:53.331450 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:53.332104 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:53.332125 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:53.332182 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:53.332239 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:53.332537 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:53.333396 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:53.333505 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:53.334073 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:53.334873 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:53.334969 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:53.335303 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:53.335721 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:53.335786 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:53.337350 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:53.337432 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:53.337508 [ 31817 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:53.338327 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:53.338419 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:53.338561 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:53.338673 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:53.340374 [ 31817 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:53.340436 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:53.340526 [ 31817 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:53.342987 [ 31817 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 12ms, ratio: 0.025641028, points per minute: 1282.0513)
2026.05.06 08:48:53.343521 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:53.343730 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:53.344249 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:53.344807 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:53.345803 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:53.346173 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:53.351301 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:53.351452 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:53.351663 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:53.351950 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.352164 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.352326 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:53.352437 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:53.352667 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:53.353702 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:53.353834 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:53.353975 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:53.354233 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.354432 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.354607 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:53.354701 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:53.354893 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:53.355990 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:53.356118 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:53.356258 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:53.356522 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.356716 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.356886 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:53.356981 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:53.357169 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:53.358193 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:53.358321 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:53.358463 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:53.358735 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.358931 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.359085 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:53.359178 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:53.359367 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:53.360479 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:53.360625 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:53.360768 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:53.361023 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.361243 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.361396 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:53.361504 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:53.361698 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:53.362929 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:53.363090 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:53.363285 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:53.363594 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.363840 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.363998 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:53.364111 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:53.364312 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:53.365088 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:53.365211 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:53.365363 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:53.365634 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.365834 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:53.365982 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:53.366074 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:53.366266 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:53.366806 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:53.367475 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:53.367938 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:53.368031 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:53.372461 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:53.373936 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:53.374612 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:53.375082 [ 31806 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:53.375409 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:53.375481 [ 31806 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:53.375821 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:53.377710 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:53.379553 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:53.380317 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:53.380390 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:53.380457 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:53.383065 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:53.385366 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:53.385537 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:53.389566 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:53.391477 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:53.391569 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:53.391728 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:53.391845 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:53.396895 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:53.397080 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:53.397569 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:53.398128 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:53.399208 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:53.399629 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:53.404128 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:53.404275 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:53.404426 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:53.404727 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.404947 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.405107 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:53.405204 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:53.405409 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:53.406363 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:53.406507 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:53.406653 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:53.406914 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.407115 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.407271 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:53.407364 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:53.407578 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:53.408669 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:53.408795 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:53.408935 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:53.409186 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.409384 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.409557 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:53.409665 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:53.409856 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:53.410956 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:53.411085 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:53.411228 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:53.411482 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.411729 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.411888 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:53.411983 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:53.412173 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:53.413250 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:53.413377 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:53.413535 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:53.413789 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.413996 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.414176 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:53.414273 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:53.414470 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:53.415440 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:53.415575 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:53.415760 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:53.416015 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.416212 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.416363 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:53.416453 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:53.416663 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:53.417401 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:53.417535 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:53.417671 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:53.417920 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.418131 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:53.418278 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:53.418369 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:53.418573 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:53.419096 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:53.419835 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:53.420247 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:53.420338 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:53.424651 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:53.425874 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:53.426446 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:53.426823 [ 31801 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:53.426991 [ 31801 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:53.545222 [ 31775 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:53.551874 [ 31775 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:53.563301 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:53.564545 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:53.581352 [ 32399 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:53.581534 [ 32399 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:53.581852 [ 32399 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.005196317 sec. (192.444 rows/sec., 4.51 KiB/sec.)
2026.05.06 08:48:53.581942 [ 32399 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:53.586135 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:53.586907 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:53.594100 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:53.594723 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:53.604746 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:53.606559 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:53.606865 [ 32280 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:54.014604 [ 31726 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.015019 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:54.015096 [ 31726 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.015346 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.015408 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:54.015771 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.017470 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.017896 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.019280 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.019529 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:54.019702 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:54.019726 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.019795 [ 31721 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.019809 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:54.019993 [ 31721 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.020023 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:54.020213 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:54.020327 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.020475 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.020532 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:54.020599 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:54.020669 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:54.021046 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:54.022095 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.022131 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.022369 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:54.023038 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:54.023151 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:54.023484 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:54.023843 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.024076 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:54.024959 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.025305 [ 31716 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.025578 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:54.025665 [ 31716 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.025708 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:54.025842 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.025857 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:54.025923 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:54.025973 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:54.026173 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.027326 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.027779 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.028817 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.029112 [ 31713 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.029465 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.029461 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:54.029805 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.031125 [ 31723 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:54.031223 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:54.031369 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:54.031561 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.033812 [ 31723 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 690ms, ratio: 0.025641028, points per minute: 22.296545)
2026.05.06 08:48:54.033807 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.034157 [ 31711 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.034436 [ 31711 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.034574 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:54.034663 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:54.034883 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.034927 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.036133 [ 31723 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:54.036295 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:54.036393 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:54.036792 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.036840 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.036976 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:54.037071 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:54.037171 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:54.037759 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:54.038578 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:54.038625 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.039036 [ 31704 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.039434 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:54.039468 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:54.039705 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:54.039818 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:54.039878 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.039919 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:54.040243 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:54.040421 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:54.040798 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:54.041186 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:54.041633 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:54.042458 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:54.042600 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:54.042611 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.042709 [ 31723 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:54.042905 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:54.043013 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:54.044426 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.045213 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:54.047704 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:54.047779 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:54.047898 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:54.048010 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:54.048056 [ 31723 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:54.048155 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:54.048277 [ 31723 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:54.050901 [ 31723 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:48:54.052747 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:54.052933 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:54.053046 [ 31723 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:54.053287 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:54.053704 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:54.054856 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:54.055244 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:54.059827 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:54.059978 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:54.060133 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:54.060419 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.060669 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.060831 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:54.060929 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:54.061135 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:54.062083 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:54.062210 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:54.062354 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:54.062636 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.062840 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.062997 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:54.063092 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:54.063288 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:54.064409 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:54.064560 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:54.064704 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:54.064972 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.065173 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.065351 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:54.065468 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:54.065697 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:54.066704 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:54.066831 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:54.066973 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:54.067227 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.067425 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.067598 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:54.067726 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:54.067916 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:54.069023 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:54.069153 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:54.069299 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:54.069579 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.069791 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.069948 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:54.070045 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:54.070245 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:54.071233 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:54.071362 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:54.071528 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:54.071818 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.072023 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.072178 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:54.072270 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:54.072463 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:54.073256 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:54.073379 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:54.073535 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:54.073790 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.073990 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:48:54.074140 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:54.074233 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:54.074428 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:54.074990 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:54.075782 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:54.076206 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:54.076299 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:54.080263 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:54.081603 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:54.082272 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:54.082742 [ 31696 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.083117 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:54.083100 [ 31696 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:54.083438 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.085307 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.086858 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:54.087598 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:54.087778 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:54.087883 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:54.090580 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:54.092909 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:54.093057 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:54.097009 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:54.099379 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:54.099456 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:54.099596 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:54.099776 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:54.104563 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:54.104772 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:54.105138 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:54.105623 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:54.106844 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:54.107229 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:54.112000 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:54.112148 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:54.112298 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:54.112600 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.112816 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.113017 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:54.113118 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:54.113325 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:54.114269 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:54.114396 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:54.114554 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:54.114861 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.115068 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.115225 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:54.115317 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:54.115531 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:54.116646 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:54.116776 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:54.116916 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:54.117196 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.117410 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.117586 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:54.117682 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:54.117893 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:54.119046 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:54.119180 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:54.119321 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:54.119596 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.119851 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.120007 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:54.120100 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:54.120289 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:54.121417 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:54.121571 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:54.121734 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:54.121998 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.122206 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.122364 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:54.122461 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:54.122688 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:54.123734 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:54.123873 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:54.124021 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:54.124282 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.124480 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.124648 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:54.124738 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:54.124932 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:54.125770 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:54.125898 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:54.126037 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:54.126289 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.126485 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:54.126649 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:54.126740 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:54.126932 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:54.127446 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:54.128170 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:54.128605 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:54.128696 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:54.132682 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:54.134059 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:54.134730 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:54.135135 [ 31693 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:54.135316 [ 31693 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:54.151386 [ 31694 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1077ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:54.215355 [ 31913 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 10573ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:54.353958 [ 32160 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 10793ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:54.384251 [ 32143 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 10818ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:54.488824 [ 32073 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:54.494946 [ 32073 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:54.504292 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:54.505402 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:54.527787 [ 32423 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:54.527935 [ 32423 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:54.528228 [ 32423 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006667619 sec. (149.979 rows/sec., 3.52 KiB/sec.)
2026.05.06 08:48:54.528314 [ 32423 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:54.533642 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:54.534273 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:54.541602 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:54.542291 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:54.554453 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:54.557505 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:54.557967 [ 32281 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:54.657367 [ 32316 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:54.657922 [ 32316 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: ::1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:54 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE72D4D06EE50
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:54.658334 [ 32316 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:54.795989 [ 32174 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.796425 [ 32174 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.796468 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:54.796786 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:54.796826 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.797061 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.799379 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.799421 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.801344 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.801369 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.801625 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:54.801758 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:54.801866 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:54.801937 [ 32088 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.802298 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:54.802371 [ 32088 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.802641 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:54.802661 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:54.802712 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:54.802672 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.802795 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:54.802958 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.803184 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:54.804474 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:54.804580 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.804859 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.805047 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:54.805168 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:54.805465 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:54.806020 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:54.806903 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.807283 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:54.807356 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:54.807419 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:54.807435 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.807836 [ 32137 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.808225 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:54.808301 [ 32137 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.808581 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:54.808592 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.808881 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.810281 [ 32083 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:54.810343 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:54.810437 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:54.810629 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.810695 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.812470 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:54.812545 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:54.812554 [ 32083 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 759ms, ratio: 0.025641028, points per minute: 20.269587)
2026.05.06 08:48:54.812696 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:54.812753 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:54.813205 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:54.813263 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:54.813311 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:54.813321 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:54.813431 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:54.813544 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:54.813654 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:54.815003 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:54.815243 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:54.815766 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:54.815878 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:54.816257 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:54.816625 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:54.816731 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:54.816759 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:54.818198 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:54.818278 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:54.818341 [ 32083 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:54.818993 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:54.821171 [ 32083 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:54.821233 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:54.821307 [ 32083 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:54.821624 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:54.821697 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:54.821834 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:54.821981 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:54.823711 [ 32083 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 11ms, ratio: 0.025641028, points per minute: 1398.6014)
2026.05.06 08:48:54.826706 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:54.826885 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:54.827375 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:54.827965 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:54.829002 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:54.829363 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:54.833997 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:54.834143 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:54.834293 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:54.834591 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.834805 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.834964 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:54.835061 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:54.835264 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:54.836247 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:54.836377 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:54.836535 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:54.836788 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.837002 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.837157 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:54.837249 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:54.837441 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:54.838508 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:54.838633 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:54.838771 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:54.839021 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.839216 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.839368 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:54.839462 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:54.839704 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:54.840694 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:54.840832 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:54.840966 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:54.841212 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.841403 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.841571 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:54.841664 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:54.841850 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:54.842908 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:54.843034 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:54.843176 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:54.843425 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.843678 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.843835 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:54.843930 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:54.844125 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:54.845098 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:54.845221 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:54.845363 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:54.845627 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.845820 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.845970 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:54.846058 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:54.846249 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:54.846987 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:54.847104 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:54.847237 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:54.847480 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.847719 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:48:54.847866 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:54.847956 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:54.848157 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:54.848683 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:54.849347 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:54.849755 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:54.849845 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:54.854150 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:54.855452 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:54.856132 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:54.856606 [ 32165 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:54.857019 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:54.857006 [ 32165 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:54.857334 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:54.859410 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:54.861695 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:54.863128 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:54.863246 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:54.863348 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:54.866577 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:54.869183 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:54.869354 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:54.873340 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:54.875520 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:54.875594 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:54.875757 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:54.875871 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:54.880573 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:54.880776 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:54.881265 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:54.881709 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:54.882985 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:54.883364 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:54.888161 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:54.888307 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:54.888457 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:54.888753 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.888965 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.889126 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:54.889222 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:54.889426 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:54.890364 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:54.890510 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:54.890652 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:54.890908 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.891122 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.891278 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:54.891371 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:54.891574 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:54.892666 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:54.892793 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:54.892942 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:54.893190 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.893384 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.893551 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:54.893645 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:54.893834 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:54.894847 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:54.894973 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:54.895109 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:54.895365 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.895589 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.895776 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:54.895869 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:54.896055 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:54.897135 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:54.897258 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:54.897397 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:54.897663 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.897865 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.898018 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:54.898112 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:54.898303 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:54.899260 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:54.899401 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:54.899557 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:54.899844 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.900042 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.900190 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:54.900277 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:54.900466 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:54.901216 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:54.901333 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:54.901466 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:54.901723 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.901918 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:54.902060 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:54.902148 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:54.902335 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:54.902868 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:54.903537 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:54.903968 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:54.904062 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:54.908513 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:54.909725 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:54.910291 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:54.910682 [ 32159 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:55.096564 [ 31640 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #5
2026.05.06 08:48:55.096699 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:48:55.097096 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:55.235882 [ 32314 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:48:55.236527 [ 32314 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: ::1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:48:55 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE72D6F82AE96
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:48:55.236982 [ 32314 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:48:55.517399 [ 32191 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.517895 [ 32191 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.517941 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:55.518209 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:55.518280 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.518514 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.520819 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.520857 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.522754 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.522753 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.523002 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:55.523149 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:55.523270 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:55.523361 [ 32175 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.523900 [ 32175 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.523883 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:55.524171 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:55.524219 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:55.524263 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.524296 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:55.524434 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:55.524514 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.525087 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:55.526137 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.526280 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.526465 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:55.527099 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:55.527313 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:55.527759 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:55.527792 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.527833 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.528212 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:55.528300 [ 31908 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.528754 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:55.528808 [ 31908 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.529112 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.529115 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:55.529435 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.529641 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:55.529786 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:55.529901 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:55.530759 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.531307 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.532511 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.532887 [ 31911 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.533034 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.533226 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:55.533455 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.535061 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.535158 [ 32125 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:55.535258 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:55.535414 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:55.536689 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:55.537399 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:55.537524 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:55.537623 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:55.537729 [ 32125 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 714ms, ratio: 0.025641028, points per minute: 21.547081)
2026.05.06 08:48:55.537903 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:55.537998 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:55.538552 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:55.538654 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:55.538767 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:55.539286 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:55.540417 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:55.540707 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:55.541352 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:55.541542 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:55.541842 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:55.542316 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:55.542807 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:55.543005 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:55.543416 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:55.543550 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:55.543695 [ 32125 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:55.547110 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:55.548914 [ 32125 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:55.549012 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:55.549125 [ 32125 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:55.549517 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:55.549589 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:55.549705 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:55.549818 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:55.551339 [ 32125 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:48:55.554064 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:55.554242 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:55.554594 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:55.555060 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:55.555779 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:55.556130 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:55.560826 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:55.560970 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:55.561117 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:55.561390 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.561621 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.561797 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:55.561893 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:55.562094 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:55.563078 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:55.563223 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:55.563380 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:55.563700 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.563921 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.564087 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:55.564188 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:55.564387 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:55.565444 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:55.565586 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:55.565726 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:55.565975 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.566182 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.566339 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:55.566432 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:55.566635 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:55.567601 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:55.567758 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:55.567897 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:55.568149 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.568342 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.568511 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:55.568604 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:55.568790 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:55.569845 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:55.569971 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:55.570127 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:55.570379 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.570601 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.570756 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:55.570851 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:55.571044 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:55.572060 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:55.572186 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:55.572328 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:55.572590 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.572786 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.572936 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:55.573026 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:55.573213 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:55.573995 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:55.574114 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:55.574251 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:55.574519 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.574715 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:48:55.574861 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:55.574952 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:55.575152 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:55.575715 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:55.576387 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:55.576805 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:55.576895 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:55.580598 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:55.581964 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:55.582658 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:55.583166 [ 32060 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.583596 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:55.583580 [ 32060 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:55.583953 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.585786 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.587460 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:55.588558 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:55.588697 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:55.588816 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:55.591637 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:55.593920 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:55.594060 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:55.598122 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:55.600288 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:55.600362 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:55.600480 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:55.600606 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:55.604924 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:55.605109 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:55.605611 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:55.606146 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:55.606959 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:55.607340 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:55.612128 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:55.612270 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:55.612415 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:55.612701 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.612911 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.613067 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:55.613175 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:55.613376 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:55.614167 [ 32053 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:55.614316 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:55.614445 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:55.614592 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:55.614803 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.614966 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.615091 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:55.615167 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:55.615325 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:55.616236 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:55.616343 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:55.616458 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:55.616681 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.616853 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.616982 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:55.617059 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:55.617213 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:55.618016 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:55.618119 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:55.618251 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:55.618451 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.618624 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.618748 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:55.618824 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:55.618977 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:55.619903 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:55.620008 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:55.620126 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:55.620298 [ 32053 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:55.620344 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.620528 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.620656 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:55.620734 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:55.620894 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:55.621694 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:55.621797 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:55.621915 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:55.622117 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.622277 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.622400 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:55.622472 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:55.622642 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:55.623279 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:55.623377 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:55.623505 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:55.623732 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.623893 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:55.624013 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:55.624085 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:55.624240 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:55.624682 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:55.625260 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:55.625601 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:55.625675 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:55.628533 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:55.629203 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:55.629742 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:55.630188 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:55.630332 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:55.630771 [ 32048 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:55.630937 [ 32048 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:55.654218 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:55.654797 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:55.801442 [ 32036 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.801914 [ 32036 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.801973 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:55.802155 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:55.802346 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.802364 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.804775 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.804877 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.806619 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.806735 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.806873 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:55.807078 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:55.807158 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:55.807199 [ 32025 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.807554 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:55.807642 [ 32025 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.807850 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:55.807911 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:55.807924 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.807923 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:55.808099 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:55.808165 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.808565 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:55.809963 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.810050 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:55.810137 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.810708 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:55.810848 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:55.811100 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:55.811475 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:55.812380 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.812426 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.812862 [ 32019 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.812880 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:55.813041 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:55.813091 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:55.813119 [ 32019 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.813160 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:55.813254 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.813313 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:55.813556 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.815020 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.815290 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.816683 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.817010 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.817013 [ 32018 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.817425 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:55.817450 [ 32018 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.817716 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.817736 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:55.818042 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.819336 [ 32033 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:55.819469 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:55.819520 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.819720 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:55.819831 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.821241 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.821613 [ 32020 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.821788 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:55.821925 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:55.822198 [ 32033 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 271ms, ratio: 0.025641028, points per minute: 56.769802)
2026.05.06 08:48:55.822206 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.822354 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:55.822426 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:55.822667 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:55.822783 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:55.822885 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:55.822964 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:55.823033 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:55.823100 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:55.823472 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:55.823969 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.824664 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:55.825259 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:55.825371 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:55.825616 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:55.825622 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:55.825673 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:55.826110 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:55.827290 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:55.827382 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:55.827451 [ 32033 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:55.827828 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:55.827972 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:55.831893 [ 32033 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:55.831968 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:55.832059 [ 32033 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:55.833660 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:55.834179 [ 32033 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 12ms, ratio: 0.025641028, points per minute: 1282.0513)
2026.05.06 08:48:55.836041 [ 32033 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:55.836450 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:55.836539 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:55.836660 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:55.836790 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:55.841446 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:55.841677 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:55.842169 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:55.842739 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:55.843581 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:55.843992 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:55.848470 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:55.848632 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:55.848778 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:55.849051 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.849262 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.849418 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:55.849529 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:55.849734 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:55.850698 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:55.850825 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:55.850966 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:55.851219 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.851431 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.851632 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:55.851727 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:55.851919 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:55.853049 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:55.853202 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:55.853347 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:55.853651 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.853850 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.854005 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:55.854099 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:55.854287 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:55.855136 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:55.855250 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:55.855365 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:55.855585 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.855773 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.855899 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:55.855978 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:55.856131 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:55.857015 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:55.857121 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:55.857240 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:55.857450 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.857636 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.857764 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:55.857843 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:55.858003 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:55.858812 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:55.858915 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:55.859034 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:55.859240 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.859398 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.859534 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:55.859631 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:55.859790 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:55.860395 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:55.860508 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:55.860619 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:55.860820 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.860980 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:48:55.861099 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:55.861183 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:55.861337 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:55.861770 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:55.862303 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:55.862647 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:55.862723 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:55.867319 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:55.868408 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:55.868950 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:55.869295 [ 32011 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:55.869622 [ 32011 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:55.869929 [ 32011 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:55.870005 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:55.870317 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:55.872211 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:55.873935 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:55.874703 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:55.874805 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:55.874904 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:55.877653 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:55.880080 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:55.880234 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:55.884250 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:55.886568 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:55.886643 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:55.886759 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:55.886865 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:55.891289 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:55.891469 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:55.891992 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:55.892485 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:55.893450 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:55.893849 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:55.897752 [ 32006 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1543ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:55.898355 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:55.898507 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:55.898649 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:55.898906 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.899107 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.899260 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:55.899350 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:55.899562 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:55.900485 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:55.900632 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:55.900768 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:55.901026 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.901219 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.901369 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:55.901457 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:55.901660 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:55.902733 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:55.902858 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:55.902995 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:55.903236 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.903427 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.903598 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:55.903722 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:55.903907 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:55.904861 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:55.904983 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:55.905116 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:55.905359 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.905563 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.905710 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:55.905800 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:55.905983 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:55.907063 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:55.907189 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:55.907328 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:55.907595 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.907861 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.908014 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:55.908106 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:55.908297 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:55.909253 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:55.909375 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:55.909533 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:55.909784 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.909979 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.910127 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:55.910220 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:55.910437 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:55.911196 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:55.911329 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:55.911463 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:55.911774 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.911973 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:48:55.912119 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:55.912222 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:55.912413 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:55.912940 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:55.913630 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:55.914022 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:55.914110 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:55.917759 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:55.919205 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:55.919959 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:55.920356 [ 32003 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:55.920536 [ 32003 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:56.119115 [ 32187 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1968ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:56.205196 [ 31987 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 4133ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:56.296053 [ 31975 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:56.296368 [ 31975 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:56.296488 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:56.296618 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:56.296879 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:56.296894 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:56.299592 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:56.299700 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:56.301368 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:56.301407 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:56.301676 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:56.301829 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:56.301947 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:56.301999 [ 31969 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:56.302325 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:56.302415 [ 31969 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:56.302642 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:56.302813 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:56.302816 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:56.302927 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:56.303042 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:56.303156 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:56.303716 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:56.304415 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:56.304852 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:56.305029 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:56.305718 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:56.305732 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:56.306056 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:56.306584 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:56.306811 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:56.307164 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:56.307155 [ 31962 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:56.307572 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:56.307660 [ 31962 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:56.307904 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:56.307977 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:56.308235 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:56.308657 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:56.308801 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:56.308915 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:56.309782 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:56.310108 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:56.311342 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:56.311740 [ 31961 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:56.311812 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:56.312025 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:56.312126 [ 31961 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:56.312307 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:56.312413 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:56.312695 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:56.314003 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:56.314117 [ 31972 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:56.314215 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:56.314355 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:56.314459 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:56.315907 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:56.316486 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:56.316597 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:56.316673 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:56.316741 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:56.316829 [ 31956 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:56.317041 [ 31972 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 481ms, ratio: 0.025641028, points per minute: 31.984648)
2026.05.06 08:48:56.317172 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:56.317216 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:56.317310 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:56.317450 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:56.317945 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:56.318035 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:56.318129 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:56.318623 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:56.319210 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:56.319701 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:56.320093 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:56.320828 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:56.321004 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:56.321025 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:56.321337 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:56.321404 [ 31954 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:56.321789 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:56.321803 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:56.322144 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:56.322281 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:56.322436 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:56.323000 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:56.323122 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:56.323228 [ 31972 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:56.324048 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:56.325936 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:56.326814 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:56.326879 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:56.326924 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:56.327006 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:56.327078 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:56.328533 [ 31972 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:56.328633 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:56.328752 [ 31972 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:56.330520 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:56.330638 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:56.330980 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:56.331483 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:56.332444 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:56.332658 [ 31972 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:48:56.332686 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:56.334872 [ 31972 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:56.335316 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:56.335409 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:56.335520 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:56.335723 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.335857 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.335957 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:56.336020 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:56.336142 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:56.336729 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:56.336812 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:56.336902 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:56.337059 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.337178 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.337273 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:56.337329 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:56.337444 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:56.338116 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:56.338199 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:56.338289 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:56.338442 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.338570 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.338665 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:56.338724 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:56.338841 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:56.339446 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:56.339535 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:56.339636 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:56.339786 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.339901 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.339994 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:56.340053 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:56.340162 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:56.340802 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:56.340880 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:56.340968 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:56.341114 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.341231 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.341330 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:56.341391 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:56.341511 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:56.342102 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:56.342178 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:56.342266 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:56.342412 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.342539 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.342631 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:56.342685 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:56.342797 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:56.343244 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:56.343317 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:56.343402 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:56.343560 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.343698 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:48:56.343785 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:56.343838 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:56.343946 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:56.344275 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:56.344702 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:56.344938 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:56.344997 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:56.348454 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:56.349121 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:56.349476 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:56.349804 [ 31951 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:56.350037 [ 31951 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:56.350261 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:56.350297 [ 31951 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:56.350480 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:56.352345 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:56.354161 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:56.354967 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:56.355083 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:56.355184 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:56.357859 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:56.360180 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:56.360326 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:56.364315 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:56.366608 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:56.366679 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:56.366795 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:56.366905 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:56.371203 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:56.371379 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:56.371896 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:56.372487 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:56.373395 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:56.373762 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:56.378448 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:56.378605 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:56.378746 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:56.379008 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.379211 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.379365 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:56.379457 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:56.379709 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:56.380621 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:56.380745 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:56.380880 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:56.381125 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.381320 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.381469 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:56.381576 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:56.381760 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:56.382773 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:56.382891 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:56.383026 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:56.383266 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.383454 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.383658 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:56.383753 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:56.383949 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:56.384892 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:56.385011 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:56.385143 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:56.385381 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.385581 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.385726 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:56.385815 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:56.385996 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:56.387020 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:56.387140 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:56.387275 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:56.387534 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.387761 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.387927 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:56.388019 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:56.388208 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:56.389143 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:56.389261 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:56.389398 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:56.389656 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.389847 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.389994 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:56.390082 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:56.390267 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:56.391005 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:56.391120 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:56.391250 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:56.391487 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.391735 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:48:56.391891 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:56.391978 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:56.392163 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:56.392671 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:56.393312 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:56.393706 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:56.393793 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:56.394221 [ 31944 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:56.397309 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:56.398605 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:56.399232 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:56.399751 [ 31941 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:56.399974 [ 31941 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:56.400224 [ 31944 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:56.408941 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:56.409921 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:56.430802 [ 32475 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:56.430891 [ 32475 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:56.431071 [ 32475 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.005726097 sec. (174.639 rows/sec., 4.09 KiB/sec.)
2026.05.06 08:48:56.431124 [ 32475 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:56.434689 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:56.435306 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:56.441897 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:56.442565 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:56.454791 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:56.458305 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:56.458887 [ 32273 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:57.046521 [ 31893 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.046835 [ 31893 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.046974 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:57.047125 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:57.047364 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.047414 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.049678 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.049768 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.051255 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.051287 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.051488 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:57.051623 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:57.051689 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:57.051879 [ 31889 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.052256 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:57.052348 [ 31889 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.052368 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:57.052484 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:57.052535 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.052630 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:57.052639 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:57.052892 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.053331 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:57.054313 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.054404 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.054778 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:57.055480 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:57.055824 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.055798 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:57.055929 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.056180 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:57.056219 [ 31882 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.056631 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:57.056679 [ 31882 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.056684 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:57.056902 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.056940 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:57.057193 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.058110 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:57.058256 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:57.058372 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:57.058668 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.058718 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.060354 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.060408 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.060923 [ 31876 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.061313 [ 31876 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.061313 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:57.061615 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:57.061696 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.061870 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.063361 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.063422 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.064454 [ 31888 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:57.064603 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:57.064780 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:57.065049 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:57.065122 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.065465 [ 31869 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.065751 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:57.065828 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:57.065835 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:57.065894 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:57.066137 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.067310 [ 31888 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 732ms, ratio: 0.025641028, points per minute: 21.017235)
2026.05.06 08:48:57.067513 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:57.067661 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:57.067800 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.068233 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:57.068345 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:57.068465 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:57.068782 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:57.069035 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:57.069266 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.070436 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:57.071085 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:57.071094 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:57.071261 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:57.071327 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:57.071669 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:57.072148 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:57.073403 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:57.073555 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:57.073669 [ 31888 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:57.075314 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:57.075749 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:57.075801 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:57.075888 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:57.075974 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:57.078996 [ 31888 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:57.079111 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:57.079235 [ 31888 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:57.079386 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:57.079509 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:57.079927 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:57.080441 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:57.081360 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:57.081615 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:57.082434 [ 31888 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:48:57.084280 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:57.084382 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:57.084448 [ 31888 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:57.084487 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:57.084686 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.084821 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.084924 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:57.084986 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:57.085114 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:57.085723 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:57.085813 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:57.085907 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:57.086073 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.086201 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.086301 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:57.086360 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:57.086483 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:57.087185 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:57.087269 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:57.087360 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:57.087526 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.087668 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.087771 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:57.087832 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:57.087952 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:57.088611 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:57.088699 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:57.088793 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:57.088958 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.089084 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.089187 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:57.089248 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:57.089367 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:57.090057 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:57.090143 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:57.090236 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:57.090396 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.090537 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.090640 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:57.090707 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:57.090827 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:57.091443 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:57.091537 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:57.091649 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:57.091811 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.091934 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.092039 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:57.092097 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:57.092214 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:57.092709 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:57.092790 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:57.092878 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:57.093033 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.093155 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:57.093254 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:57.093311 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:57.093429 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:57.093798 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:57.094257 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:57.094525 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:57.094588 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:57.097847 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:57.098570 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.098942 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.099259 [ 32089 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:57.099656 [ 32089 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.099968 [ 32089 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:57.100028 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:57.100326 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.102252 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.103969 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:57.104655 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:57.104731 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:57.104799 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:57.107468 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:57.109880 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:57.110036 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:57.114043 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:57.114692 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:57.114751 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:57.114835 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:57.114917 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:57.118457 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:57.118592 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:57.119049 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:57.119693 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:57.120639 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:57.120929 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:57.123563 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:57.123700 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:57.123815 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:57.124017 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.124167 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.124284 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:57.124356 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:57.124508 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:57.125162 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:57.125265 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:57.125366 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:57.125559 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.125701 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.125811 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:57.125876 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:57.126004 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:57.126761 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:57.126850 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:57.126942 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:57.127102 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.127226 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.127325 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:57.127387 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:57.127515 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:57.128197 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:57.128289 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:57.128384 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:57.128567 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.128700 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.128802 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:57.128863 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:57.128980 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:57.129681 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:57.129766 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:57.129860 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:57.130021 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.130149 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.130248 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:57.130320 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:57.130443 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:57.131087 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:57.131173 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:57.131266 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:57.131429 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.131568 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.131693 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:57.131751 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:57.131867 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:57.132340 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:57.132419 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:57.132522 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:57.132676 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.132796 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:48:57.132893 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:57.132950 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:57.133063 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:57.133419 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:57.133892 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:57.134169 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:57.134239 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:57.137286 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:57.137995 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.138362 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.138772 [ 32084 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:57.139057 [ 32084 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:57.256893 [ 31852 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:57.263034 [ 31852 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:57.267697 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:57.268327 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:57.281109 [ 32452 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:57.281189 [ 32452 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:57.281341 [ 32452 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.004376515 sec. (228.492 rows/sec., 5.36 KiB/sec.)
2026.05.06 08:48:57.281397 [ 32452 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:57.285075 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:57.285678 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:57.292246 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:57.292974 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:57.303050 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.305471 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.305764 [ 32278 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:57.607830 [ 31832 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1488ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:57.780059 [ 31824 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:57.786177 [ 31824 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:57.795220 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:57.796291 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:57.818881 [ 32467 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:57.819029 [ 32467 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:57.819355 [ 32467 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006612878 sec. (151.220 rows/sec., 3.54 KiB/sec.)
2026.05.06 08:48:57.819483 [ 32467 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:57.825650 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:57.826375 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:57.833030 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:57.833841 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:57.842591 [ 31814 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.843032 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:57.843084 [ 31814 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.843356 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.843387 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:57.843790 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.844551 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.845831 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.845957 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.847424 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.847738 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.847839 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.847920 [ 32275 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:57.847931 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:57.848060 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:57.848130 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:57.848563 [ 31817 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.848727 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:57.848799 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:57.848882 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:57.849019 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:57.849121 [ 31817 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.849283 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:57.849418 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:57.849478 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.849678 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.850806 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:57.851444 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:57.851562 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.851644 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.851592 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:57.852140 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:57.852685 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:57.854023 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:57.854117 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:57.854183 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:57.854219 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.854375 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.854815 [ 31803 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.855178 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:57.855238 [ 31803 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.855513 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:57.855533 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.855827 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.857041 [ 31809 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:57.857109 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:57.857218 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:57.857594 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.857660 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.859538 [ 31809 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 775ms, ratio: 0.025641028, points per minute: 19.851118)
2026.05.06 08:48:57.859531 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:57.859582 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.859999 [ 31798 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.860506 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:57.860644 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:57.860777 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:57.860899 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:57.860895 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.861590 [ 31809 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:57.861693 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:57.861758 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:57.862258 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:57.862325 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:57.862392 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:57.862750 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:57.862761 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.863726 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:57.864182 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:57.864641 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:57.864775 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:57.864902 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:57.865233 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:57.865732 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:57.866010 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:57.866161 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:57.866825 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:57.866920 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:57.866986 [ 31809 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:57.870165 [ 31809 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:57.870210 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:57.870232 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:57.870328 [ 31809 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:57.872572 [ 31809 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 11ms, ratio: 0.025641028, points per minute: 1398.6014)
2026.05.06 08:48:57.872827 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:57.872909 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:57.873038 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:57.873172 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:57.874400 [ 31809 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:57.877997 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:57.878205 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:57.878769 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:57.879312 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:57.880363 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:57.880808 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:57.885416 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:57.885592 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:57.885752 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:57.886058 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.886281 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.886446 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:57.886560 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:57.886769 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:57.887778 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:57.887915 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:57.888061 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:57.888327 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.888573 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.888750 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:57.888845 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:57.889042 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:57.890156 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:57.890286 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:57.890435 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:57.890717 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.890920 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.891082 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:57.891198 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:57.891394 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:57.892482 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:57.892636 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:57.892780 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:57.893038 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.893256 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.893414 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:57.893527 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:57.893719 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:57.894825 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:57.894958 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:57.895105 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:57.895401 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.895723 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.895940 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:57.896078 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:57.896331 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:57.897391 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:57.897540 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:57.897690 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:57.897967 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.898170 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.898324 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:57.898416 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:57.898626 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:57.899382 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:57.899518 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:57.899754 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:57.900035 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.900240 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:48:57.900392 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:57.900486 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:57.900703 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:57.901237 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:57.901972 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:57.902391 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:57.902483 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:57.906338 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:57.907785 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.908449 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.908933 [ 31792 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:57.909282 [ 31792 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:57.909626 [ 31792 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:57.909679 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:57.910015 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:57.912113 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:57.914015 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:57.914978 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:57.915090 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:57.915193 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:57.917987 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:57.920288 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:57.920433 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:57.924405 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:57.926657 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:57.926731 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:57.926851 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:57.926966 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:57.931764 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:57.931954 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:57.932616 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:57.933310 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:57.934472 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:57.934878 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:57.939763 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:57.939910 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:57.940061 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:57.940341 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.940570 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.940729 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:57.940826 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:57.941029 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:57.941968 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:57.942093 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:57.942233 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:57.942484 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.942699 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.942882 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:57.943003 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:57.943266 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:57.944442 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:57.944596 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:57.944737 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:57.944992 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.945190 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.945346 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:57.945438 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:57.945644 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:57.946621 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:57.946744 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:57.946881 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:57.947124 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.947331 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.947566 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:57.947742 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:57.947939 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:57.949039 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:57.949164 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:57.949306 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:57.949581 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.949784 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.949992 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:57.950109 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:57.950313 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:57.951347 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:57.951474 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:57.951704 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:57.952008 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.952231 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.952388 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:57.952488 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:57.952737 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:57.953549 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:57.953721 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:57.953911 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:57.954246 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.954527 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:48:57.954695 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:57.954788 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:57.954983 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:57.955523 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:57.956228 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:57.956647 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:57.956736 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:57.960561 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:57.962053 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.962760 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:57.963157 [ 31782 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:58.494270 [ 31742 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:58.501137 [ 31742 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:58.510599 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:58.511726 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:58.535590 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:58.535816 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:58.536133 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006986928 sec. (143.124 rows/sec., 3.35 KiB/sec.)
2026.05.06 08:48:58.536227 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:58.540325 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:58.540987 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:58.547857 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:58.548675 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:58.564825 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:58.569433 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:58.570140 [ 32272 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:58.885841 [ 31710 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:58.889937 [ 31710 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:58.899377 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:58.900506 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:58.909868 [ 31704 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 4525ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:58.923455 [ 32347 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:58.923657 [ 32347 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:58.923977 [ 32347 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006894258 sec. (145.048 rows/sec., 3.40 KiB/sec.)
2026.05.06 08:48:58.924069 [ 32347 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:58.928379 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:58.929049 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:58.936247 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:58.936935 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:58.950234 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:58.953719 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:58.954196 [ 32280 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:48:59.017893 [ 31700 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.018341 [ 31700 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.018419 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:59.018750 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:59.018851 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.019160 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.021516 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.021612 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.023266 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.023336 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.023537 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:59.023787 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:59.023916 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:59.023933 [ 31695 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.024525 [ 31695 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.024558 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:59.024725 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:59.024837 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.024841 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:59.024973 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:59.024998 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:59.025406 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.025739 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:59.026949 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.027436 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:59.027460 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.028247 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:59.028448 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:59.028843 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:59.028950 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.029366 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:59.029390 [ 31691 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.029841 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:59.030220 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.030328 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.030795 [ 31688 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.030939 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:59.031101 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:59.031184 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:59.031218 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:59.031534 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.032197 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.033577 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.033973 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.035666 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.036110 [ 31687 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.036445 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:59.036523 [ 31687 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.036653 [ 31696 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:59.036762 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:59.036812 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:59.036812 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.036944 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:59.037095 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.039171 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.039221 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.039812 [ 31696 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1165ms, ratio: 0.025641028, points per minute: 13.205679)
2026.05.06 08:48:59.040015 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:59.040117 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:59.040737 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:59.040834 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:59.040937 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:59.040990 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:59.041029 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.041468 [ 31898 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.041517 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:59.041826 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:59.041994 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:59.042104 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.042104 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:59.042265 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:59.042887 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:59.043661 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:59.043853 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:59.043861 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.044242 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:59.044693 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:59.045093 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:59.046034 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:59.046164 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:59.046273 [ 31696 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:59.046673 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.047087 [ 31892 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.047355 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:59.047514 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:59.047520 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:59.047912 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.049770 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.051671 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.052806 [ 31696 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:59.052904 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:59.053026 [ 31696 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:59.054237 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:59.055966 [ 31696 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:48:59.056568 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:59.056641 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:59.056757 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:59.056868 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:59.058301 [ 31696 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:59.061799 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:59.061977 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:59.062592 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:59.063241 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:59.064335 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:59.064735 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:59.069232 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:59.069375 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:59.069542 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:59.069830 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.070042 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.070198 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:59.070293 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:59.070510 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:59.071421 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:59.071566 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:59.071737 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:59.072001 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.072242 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.072402 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:59.072510 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:59.072702 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:59.073773 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:59.073898 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:59.074039 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:59.074291 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.074487 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.074659 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:59.074761 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:59.074964 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:59.075957 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:59.076085 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:59.076223 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:59.076467 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.076675 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.076827 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:59.076918 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:59.077107 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:59.078180 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:59.078306 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:59.078448 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:59.078713 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.078917 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.079086 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:59.079181 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:59.079376 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:59.080388 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:59.080535 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:59.080681 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:59.080937 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.081138 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.081292 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:59.081384 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:59.081590 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:59.082345 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:59.082467 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:59.082622 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:59.082885 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.083099 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.083249 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:59.083341 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:59.083552 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:59.084173 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:59.084893 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:59.085318 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:59.085423 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:59.090317 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:59.091720 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:59.092385 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:59.092937 [ 32078 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:59.093433 [ 32078 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.093919 [ 32078 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:59.093948 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:59.094307 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.096667 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.098629 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:59.099602 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:59.099757 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:59.099862 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:59.102765 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:59.105194 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:59.105361 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:59.109524 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:59.111935 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:59.112019 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:59.112140 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:59.112257 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:59.116823 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:59.117007 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:59.117614 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:59.118310 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:59.119451 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:59.119878 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:59.124398 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:59.124560 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:59.124713 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:59.124999 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.125222 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.125384 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:59.125482 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:59.125707 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:59.126662 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:59.126791 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:59.126940 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:59.127209 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.127422 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.127648 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:59.127750 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:59.127957 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:59.129044 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:59.129177 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:59.129326 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:59.129606 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.129817 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.129980 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:59.130079 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:59.130304 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:59.131333 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:59.131464 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:59.131657 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:59.131936 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.132144 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.132307 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:59.132405 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:59.132623 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:59.133729 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:59.133861 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:59.134013 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:59.134280 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.134515 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.134693 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:59.134792 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:59.134997 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:59.136029 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:59.136162 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:59.136312 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:59.136588 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.136797 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.136974 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:59.137070 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:59.137272 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:59.138066 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:59.138191 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:59.138334 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:59.138612 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.138836 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:48:59.138990 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:59.139085 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:59.139296 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:59.139919 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:59.140629 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:59.141043 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:59.141137 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:59.146485 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:59.147214 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:59.147584 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:59.148118 [ 32130 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:59.148366 [ 32130 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:59.177321 [ 32160 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1570ms, ratio: 0, points per minute: 0)
2026.05.06 08:48:59.353955 [ 32061 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.354356 [ 32061 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.354488 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:59.354694 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:59.354959 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.355046 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.357957 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.358030 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.359835 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.359901 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.360050 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:48:59.360179 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:48:59.360261 [ 32079 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.360281 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:59.360585 [ 32079 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.360775 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:59.360768 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:59.360951 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:59.361035 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.361056 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:59.361158 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:48:59.361228 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.361710 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:59.362873 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.363135 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:48:59.363168 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.363970 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:59.364172 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:59.364597 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:59.364686 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.365123 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:59.366101 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.366546 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:48:59.366568 [ 31904 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.366710 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:59.366837 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:48:59.367027 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:48:59.367033 [ 31904 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.367533 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.367541 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:59.367920 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.371268 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.371586 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.373142 [ 32073 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:59.373254 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:48:59.373390 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:59.375302 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:59.375658 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:48:59.376119 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:59.376192 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:59.376254 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:59.376375 [ 32073 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 318ms, ratio: 0.025641028, points per minute: 48.3793)
2026.05.06 08:48:59.376577 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:48:59.376663 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:48:59.377251 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:48:59.377336 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:48:59.377424 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:48:59.377889 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:48:59.379099 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:59.379334 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:48:59.380020 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:48:59.380192 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:48:59.380536 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:48:59.381021 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:59.381738 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:59.381946 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:48:59.382430 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:48:59.382569 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:59.382667 [ 32073 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:48:59.386303 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:59.386727 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:59.386783 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:59.386872 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:59.386944 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:59.387881 [ 32073 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:48:59.387977 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:48:59.388094 [ 32073 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:48:59.390227 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:48:59.390332 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:48:59.390833 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:59.391488 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:59.392601 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:48:59.392742 [ 32073 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:48:59.392832 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:59.395289 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:48:59.395383 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:48:59.395481 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:59.395687 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.395818 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.395918 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:59.395978 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:59.396100 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:59.396686 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:48:59.396767 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:48:59.396854 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:59.397005 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.397123 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.397220 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:59.397278 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:59.397391 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:59.398066 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:48:59.398148 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:48:59.398236 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:59.398387 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.398513 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.398611 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:59.398680 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:59.398791 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:59.399399 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:48:59.399479 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:48:59.399580 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:59.399742 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.399857 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.399947 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:59.400003 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:59.400111 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:59.400750 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:48:59.400830 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:48:59.400918 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:59.401068 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.401189 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.401290 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:59.401349 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:59.401461 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:59.402056 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:48:59.402134 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:48:59.402223 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:59.402375 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.402501 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.402597 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:59.402650 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:59.402758 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:59.403210 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:48:59.403285 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:48:59.403367 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:59.403525 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.403662 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:48:59.403754 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:59.403811 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:59.403924 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:59.404244 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:48:59.404649 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:59.404891 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:59.404949 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:48:59.408209 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:48:59.408946 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:59.409316 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:59.409797 [ 31885 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:59.410229 [ 31885 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:48:59.410616 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:48:59.410706 [ 31885 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:59.410945 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:48:59.412619 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:48:59.414111 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:48:59.414725 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:48:59.414775 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:48:59.414823 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:48:59.417466 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:48:59.419782 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:48:59.419933 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:48:59.424009 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:48:59.424044 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:48:59.424077 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:48:59.424129 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:48:59.424200 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:48:59.426552 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:48:59.426632 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:48:59.427039 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:48:59.427718 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:48:59.428768 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:48:59.428944 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:48:59.430795 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:48:59.430865 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:48:59.430934 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:48:59.431055 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.431150 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.431219 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:48:59.431261 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:48:59.431351 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:48:59.431801 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:48:59.431862 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:48:59.431928 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:48:59.432059 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.432148 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.432216 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:48:59.432257 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:48:59.432342 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:48:59.432823 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:48:59.432879 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:48:59.432943 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:59.433055 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.433142 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.433211 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:59.433261 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:59.433344 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:59.433781 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:48:59.433838 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:48:59.433902 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:48:59.434014 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.434097 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.434164 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:48:59.434203 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:48:59.434284 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:48:59.434783 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:48:59.434842 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:48:59.434906 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:48:59.435018 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.435108 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.435185 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:48:59.435227 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:48:59.435313 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:48:59.435774 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:48:59.435832 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:48:59.435898 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:48:59.436030 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.436123 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.436193 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:48:59.436233 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:48:59.436321 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:48:59.436672 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:48:59.436727 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:48:59.436791 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:48:59.436904 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.437000 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:48:59.437067 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:48:59.437108 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:48:59.437193 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:48:59.437448 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:48:59.437806 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:48:59.437990 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:48:59.438031 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:48:59.441088 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:48:59.441656 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:59.441934 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:59.442239 [ 31910 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:59.442436 [ 31910 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:48:59.597710 [ 32088 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:59.604714 [ 32088 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:48:59.614032 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:48:59.615085 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:48:59.636974 [ 32401 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:48:59.637089 [ 32401 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:48:59.637300 [ 32401 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006147528 sec. (162.667 rows/sec., 3.81 KiB/sec.)
2026.05.06 08:48:59.637360 [ 32401 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:48:59.642993 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:48:59.643753 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:48:59.650704 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:48:59.651426 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:48:59.663396 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:59.666506 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:48:59.666989 [ 32281 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:00.177781 [ 32181 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.178225 [ 32181 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.178291 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:00.179981 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.180044 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.180340 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.182485 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.182587 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.184221 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.184283 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.184453 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:00.184619 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:00.184722 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:00.184791 [ 32120 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.185228 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:00.185300 [ 32120 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.185462 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:00.185567 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.185588 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:00.185646 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.185718 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:00.185906 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.186344 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:00.187279 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.187684 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.187795 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:00.188528 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:00.188748 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:00.188885 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.189221 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:00.189457 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.189720 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:00.189841 [ 32162 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.190176 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:00.190224 [ 32162 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.190454 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.190561 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.190942 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.191406 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:00.191560 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:00.191694 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:00.192312 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.192740 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.194009 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.194546 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.194513 [ 32149 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.194957 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:00.195027 [ 32149 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.195302 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.195306 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.195583 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.196849 [ 31928 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:00.196949 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:00.197000 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.197089 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.197104 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:00.198715 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:00.198801 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.199184 [ 32169 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.199577 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:00.199580 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.199590 [ 31928 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 807ms, ratio: 0.025641028, points per minute: 19.063961)
2026.05.06 08:49:00.199710 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:00.199809 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:00.199829 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:00.199913 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:00.199920 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.200564 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:00.200648 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:00.200748 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:00.201193 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:00.201853 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.202543 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:00.202586 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:00.203181 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:00.203349 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:00.203556 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.203697 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:00.203944 [ 32147 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.204199 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:00.204298 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.204642 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.204885 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:00.205027 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:00.205445 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:00.205578 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:00.205677 [ 31928 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:00.206244 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.207833 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.209025 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:00.210784 [ 31928 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:00.210871 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:00.210974 [ 31928 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:00.211255 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:00.211336 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:00.211451 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:00.211578 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:00.213283 [ 31928 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:00.215706 [ 31928 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:00.216447 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:00.216654 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:00.217128 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:00.217670 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:00.218783 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:00.219143 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:00.223919 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:00.224063 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:00.224213 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:00.224508 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.224722 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.224894 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:00.224990 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:00.225195 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:00.226125 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:00.226250 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:00.226389 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:00.226658 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.226857 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.227010 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:00.227101 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:00.227292 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:00.228402 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:00.228550 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:00.228688 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:00.228950 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.229157 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.229311 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:00.229405 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:00.229637 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:00.230612 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:00.230735 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:00.230871 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:00.231118 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.231313 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.231464 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:00.231576 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:00.231791 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:00.232873 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:00.232998 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:00.233153 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:00.233403 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.233622 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.233777 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:00.233873 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:00.234071 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:00.235052 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:00.235177 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:00.235320 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:00.235586 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.235826 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.235980 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:00.236070 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:00.236264 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:00.237045 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:00.237168 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:00.237303 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:00.237568 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.237767 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:00.237915 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:00.238006 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:00.238199 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:00.238743 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:00.239415 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:00.239872 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:00.239968 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:00.243434 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:00.244980 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:00.245682 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:00.246235 [ 32135 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:00.246725 [ 32135 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.247114 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.247117 [ 32135 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:00.247405 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.249258 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.250870 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:00.251722 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:00.251845 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:00.251943 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:00.254866 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:00.257329 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:00.257508 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:00.261607 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:00.263710 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:00.263787 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:00.263902 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:00.264012 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:00.268655 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:00.268833 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:00.269352 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:00.269951 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:00.271204 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:00.271586 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:00.276083 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:00.276225 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:00.276366 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:00.276646 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.276856 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.277024 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:00.277130 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:00.277333 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:00.278246 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:00.278370 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:00.278527 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:00.278800 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.279020 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.279214 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:00.279333 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:00.279590 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:00.281046 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:00.281210 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:00.281381 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:00.281703 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.281961 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.282154 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:00.282273 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:00.282528 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:00.283957 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:00.284138 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:00.284328 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:00.284685 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.284953 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.285165 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:00.285295 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:00.285571 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:00.287045 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:00.287221 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:00.287440 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:00.287881 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.288165 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.288379 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:00.288532 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:00.288763 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:00.289934 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:00.290082 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:00.290245 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:00.290546 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.290771 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.290945 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:00.291052 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:00.291265 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:00.292265 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:00.292412 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:00.292589 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:00.292873 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.293102 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:00.293300 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:00.293431 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:00.293663 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:00.294296 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:00.295130 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:00.295654 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:00.295768 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:00.299673 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:00.301026 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:00.301753 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:00.302192 [ 32163 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:00.433166 [ 31911 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:00.439049 [ 31911 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:00.447946 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:00.448962 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:00.470808 [ 32371 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:00.470999 [ 32371 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:00.471321 [ 32371 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006866128 sec. (145.642 rows/sec., 3.41 KiB/sec.)
2026.05.06 08:49:00.471414 [ 32371 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:00.475466 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:00.476118 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:00.482831 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:00.483426 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:00.495464 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:00.498298 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:00.498759 [ 32271 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:00.713898 [ 32038 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.714370 [ 32038 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.714448 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:00.714820 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.714841 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.715255 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.717052 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.717198 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.718647 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.718892 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:00.719101 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:00.719137 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.719207 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:00.719306 [ 32048 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.719780 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:00.719846 [ 32048 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.719929 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:00.720032 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:00.720118 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.720159 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:00.720269 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.720684 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.720746 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:00.721639 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.722089 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:00.722417 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.722767 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:00.722973 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:00.723274 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.723433 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:00.723691 [ 32043 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.724011 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:00.724027 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:00.724322 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.725083 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.725354 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:00.725482 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:00.725567 [ 32052 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.725604 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:00.725954 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.725964 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.726232 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.727450 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.727826 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.727902 [ 32039 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.728274 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:00.728523 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.729525 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.729909 [ 32056 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.730074 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.730285 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.730568 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.730737 [ 32057 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:00.730835 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:00.730978 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:00.731651 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:00.732409 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.732427 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:00.732576 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:00.732674 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:00.733242 [ 32057 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 518ms, ratio: 0.025641028, points per minute: 29.700031)
2026.05.06 08:49:00.733408 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:00.733507 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:00.733986 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:00.733985 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:00.734070 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:00.734156 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:00.734626 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:00.735344 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:00.735841 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:00.736449 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:00.736633 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:00.736899 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:00.737283 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:00.737681 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:00.737820 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:00.738638 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:00.738749 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:00.738847 [ 32057 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:00.741760 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:00.744033 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:00.744054 [ 32057 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:00.744107 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:00.744142 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:00.744222 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:00.744246 [ 32057 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:00.744333 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:00.746508 [ 32057 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 13ms, ratio: 0.025641028, points per minute: 1183.432)
2026.05.06 08:49:00.748315 [ 32057 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:00.749005 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:00.749183 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:00.749543 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:00.750108 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:00.751243 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:00.751644 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:00.756348 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:00.756489 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:00.756652 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:00.756924 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.757134 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.757290 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:00.757384 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:00.757601 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:00.758530 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:00.758653 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:00.758804 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:00.759056 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.759251 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.759401 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:00.759507 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:00.759741 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:00.760784 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:00.760904 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:00.761040 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:00.761300 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.761536 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.761697 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:00.761790 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:00.761988 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:00.763043 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:00.763174 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:00.763314 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:00.763587 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.763818 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.763970 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:00.764062 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:00.764248 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:00.765314 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:00.765437 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:00.765599 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:00.765864 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.766068 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.766220 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:00.766315 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:00.766535 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:00.767516 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:00.767669 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:00.767819 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:00.768069 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.768262 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.768407 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:00.768509 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:00.768697 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:00.769431 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:00.769573 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:00.769705 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:00.769947 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.770139 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:00.770294 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:00.770381 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:00.770582 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:00.771087 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:00.771767 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:00.772156 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:00.772246 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:00.776041 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:00.777539 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:00.778223 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:00.778644 [ 32051 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:00.778997 [ 32051 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:00.779365 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:00.779394 [ 32051 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:00.779736 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:00.781315 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:00.782807 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:00.783563 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:00.783699 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:00.783792 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:00.785375 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:00.786592 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:00.786677 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:00.788775 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:00.791120 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:00.791187 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:00.791310 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:00.791412 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:00.796045 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:00.796219 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:00.796657 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:00.797221 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:00.798296 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:00.798651 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:00.803107 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:00.803245 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:00.803383 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:00.803685 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.803895 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.804047 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:00.804145 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:00.804334 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:00.805222 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:00.805342 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:00.805507 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:00.805750 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.805942 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.806096 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:00.806185 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:00.806368 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:00.807400 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:00.807538 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:00.807704 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:00.807946 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.808151 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.808297 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:00.808385 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:00.808581 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:00.809520 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:00.809638 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:00.809769 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:00.810005 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.810194 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.810339 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:00.810444 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:00.810671 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:00.811758 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:00.811881 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:00.812013 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:00.812254 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.812446 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.812614 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:00.812703 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:00.812906 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:00.813813 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:00.813911 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:00.814023 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:00.814221 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.814375 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.814508 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:00.814578 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:00.814729 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:00.815318 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:00.815411 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:00.815529 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:00.815757 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.815913 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:00.816038 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:00.816109 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:00.816278 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:00.816698 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:00.817207 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:00.817527 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:00.817598 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:00.820683 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:00.821722 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:00.822231 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:00.822758 [ 32036 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:00.823048 [ 32036 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:00.914286 [ 32037 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 10509ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:01.139393 [ 32008 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1962ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:01.221726 [ 32011 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:01.222150 [ 32011 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:01.222233 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:01.222484 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:01.222666 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:01.222919 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:01.224784 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:01.224875 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:01.226536 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:01.226599 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:01.226763 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:01.226945 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:01.227069 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:01.227121 [ 32017 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:01.227539 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:01.227640 [ 32017 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:01.227690 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:01.227793 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:01.227920 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:01.227916 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:01.228088 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:01.228534 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:01.228697 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:01.229425 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:01.230139 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:01.230144 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:01.230797 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:01.230854 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:01.231078 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:01.231436 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:01.231977 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:01.232842 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:01.233132 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:01.233128 [ 32000 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:01.233295 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:01.233408 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:01.233518 [ 32000 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:01.233577 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:01.233756 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:01.233929 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:01.234069 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:01.235711 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:01.235811 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:01.237424 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:01.237593 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:01.238000 [ 32009 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:01.238278 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:01.238363 [ 32009 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:01.238567 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:01.238612 [ 32003 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:01.238732 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:01.238823 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:01.238887 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:01.239177 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:01.240354 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:01.241029 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:01.241288 [ 32003 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 493ms, ratio: 0.025641028, points per minute: 31.20612)
2026.05.06 08:49:01.241465 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:01.241570 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:01.241861 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:01.242151 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:01.242242 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:01.242338 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:01.242549 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:01.242626 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:01.242696 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:01.242776 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:01.242860 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:01.244264 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:01.244921 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:01.245096 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:01.245401 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:01.245784 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:01.245950 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:01.247053 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:01.247176 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:01.247282 [ 32003 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:01.248402 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:01.248615 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:01.253855 [ 32003 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:01.253955 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:01.254071 [ 32003 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:01.255651 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:01.255719 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:01.255773 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:01.255854 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:01.255940 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:01.256798 [ 32003 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:01.259279 [ 32003 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:01.259378 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:01.259503 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:01.259935 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:01.260602 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:01.261418 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:01.261685 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:01.264291 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:01.264394 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:01.264509 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:01.264697 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.264838 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.264943 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:01.265008 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:01.265140 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:01.265742 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:01.265829 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:01.265924 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:01.266089 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.266216 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.266319 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:01.266381 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:01.266509 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:01.267199 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:01.267281 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:01.267375 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:01.267546 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.267690 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.267794 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:01.267857 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:01.267970 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:01.268611 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:01.268695 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:01.268784 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:01.268937 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.269055 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.269159 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:01.269217 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:01.269328 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:01.269995 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:01.270076 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:01.270167 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:01.270319 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.270444 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.270551 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:01.270612 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:01.270726 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:01.271324 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:01.271404 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:01.271503 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:01.271674 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.271795 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.271896 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:01.271953 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:01.272066 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:01.272534 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:01.272611 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:01.272696 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:01.272845 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.272963 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:01.273050 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:01.273104 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:01.273213 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:01.273569 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:01.273982 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:01.274233 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:01.274299 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:01.277417 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:01.278034 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:01.278324 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:01.278730 [ 31986 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:01.279102 [ 31986 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:01.279438 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:01.279482 [ 31986 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:01.279744 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:01.281562 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:01.283325 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:01.284242 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:01.284314 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:01.284378 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:01.286984 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:01.289299 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:01.289448 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:01.293691 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:01.293746 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:01.293792 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:01.293875 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:01.293949 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:01.297319 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:01.297442 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:01.297931 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:01.298549 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:01.299418 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:01.299711 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:01.302115 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:01.302214 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:01.302316 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:01.302512 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.302652 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.302753 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:01.302816 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:01.302942 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:01.303525 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:01.303624 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:01.303718 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:01.303896 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.304023 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.304122 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:01.304182 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:01.304301 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:01.304997 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:01.305080 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:01.305172 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:01.305330 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.305456 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.305569 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:01.305631 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:01.305750 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:01.306368 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:01.306450 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:01.306553 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:01.306711 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.306833 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.306930 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:01.306997 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:01.307113 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:01.307811 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:01.307895 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:01.307985 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:01.308142 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.308269 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.308367 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:01.308428 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:01.308556 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:01.309193 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:01.309306 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:01.309405 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:01.309584 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.309710 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.309816 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:01.309874 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:01.309988 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:01.310455 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:01.310547 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:01.310635 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:01.310790 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.310911 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:01.311002 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:01.311060 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:01.311172 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:01.311557 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:01.311974 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:01.312223 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:01.312281 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:01.316192 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:01.316676 [ 32318 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:01.316864 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:01.317205 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:01.317228 [ 32318 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: ::1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:01 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE72ED9F6CA5A
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:01.317575 [ 31984 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:01.317686 [ 32318 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:01.317784 [ 31984 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:01.351042 [ 31979 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:01.357074 [ 31979 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:01.366125 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:01.367161 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:01.390130 [ 32327 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:01.390310 [ 32327 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:01.390717 [ 32327 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007308259 sec. (136.831 rows/sec., 3.21 KiB/sec.)
2026.05.06 08:49:01.390884 [ 32327 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:01.395004 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:01.395693 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:01.402738 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:01.403389 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:01.415883 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:01.418879 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:01.419350 [ 32284 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:01.934578 [ 31972 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9130ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:02.096537 [ 31945 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:02.102644 [ 31945 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:02.111671 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:02.112723 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:02.135285 [ 32427 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:02.135423 [ 32427 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:02.135773 [ 32427 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006442068 sec. (155.230 rows/sec., 3.64 KiB/sec.)
2026.05.06 08:49:02.135881 [ 32427 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:02.139885 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:02.140560 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:02.147375 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:02.148060 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:02.160130 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.163011 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.163481 [ 32273 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:02.249305 [ 31929 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.249792 [ 31929 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.249812 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:02.250105 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.250098 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:02.250539 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.252415 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.252530 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.254024 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.254156 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.254256 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:02.254462 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:02.254614 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:02.254623 [ 31926 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.254967 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:02.255022 [ 31926 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.255276 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.255325 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:02.255411 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:02.255535 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:02.255662 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.255689 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:02.256443 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:02.257260 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.257579 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.258024 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:02.258740 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:02.258978 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:02.259192 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.259359 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:02.259567 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.259683 [ 32131 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.259980 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:02.260118 [ 32131 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.260135 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:02.260396 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:02.260446 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.260688 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.261564 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:02.261741 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:02.261857 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:02.262464 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.262617 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.264513 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.264641 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.264895 [ 31921 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.265231 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:02.265303 [ 31921 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.265610 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.265641 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:02.265905 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.267776 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.267867 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.268532 [ 31930 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:02.268660 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:02.268843 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:02.269729 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:02.269810 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.270174 [ 31980 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.270535 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:02.270577 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:02.270659 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:02.270728 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:02.270802 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.271343 [ 31930 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1012ms, ratio: 0.025641028, points per minute: 15.202189)
2026.05.06 08:49:02.271542 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:02.271690 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:02.272249 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:02.272369 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:02.272468 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:02.272487 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.273030 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:02.273589 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:02.274251 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.274277 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:02.274977 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:02.275156 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:02.275554 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:02.276039 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:02.276042 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:02.276209 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:02.277580 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:02.277713 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:02.277819 [ 31930 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:02.280733 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:02.280807 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:02.280858 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:02.280949 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:02.281030 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:02.283135 [ 31930 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:02.283234 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:02.283370 [ 31930 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:02.284584 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:02.284704 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:02.285105 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:02.285623 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:02.286482 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:02.286737 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:02.287264 [ 31930 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:49:02.289519 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:02.289624 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:02.289666 [ 31930 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:02.289728 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:02.289923 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.290066 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.290178 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:02.290245 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:02.290375 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:02.290990 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:02.291091 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:02.291184 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:02.291356 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.291505 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.291627 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:02.291694 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:02.291820 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:02.292545 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:02.292633 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:02.292727 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:02.292886 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.293016 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.293118 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:02.293182 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:02.293297 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:02.293937 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:02.294019 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:02.294112 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:02.294278 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.294401 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.294510 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:02.294569 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:02.294689 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:02.295352 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:02.295442 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:02.295554 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:02.295749 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.295891 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.295999 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:02.296066 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:02.296205 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:02.296833 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:02.296918 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:02.297016 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:02.297176 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.297281 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.297360 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:02.297406 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:02.297523 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:02.297948 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:02.298028 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:02.298117 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:02.298276 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.298400 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:02.298514 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:02.298577 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:02.298699 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:02.299073 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:02.299544 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:02.299828 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:02.299892 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:02.303004 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:02.303773 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.304119 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.304513 [ 31909 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:02.304862 [ 31909 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.305194 [ 31909 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:02.305229 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:02.305597 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.307566 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.309613 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:02.310475 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:02.310639 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:02.310760 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:02.313516 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:02.315821 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:02.315965 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:02.319870 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:02.322244 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:02.322318 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:02.322450 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:02.322583 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:02.327573 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:02.327794 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:02.328336 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:02.328904 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:02.329917 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:02.330293 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:02.335468 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:02.335676 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:02.335840 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:02.336163 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.336384 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.336561 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:02.336658 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:02.336886 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:02.337880 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:02.338038 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:02.338224 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:02.338534 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.338740 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.338943 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:02.339051 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:02.339248 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:02.340427 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:02.340580 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:02.340721 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:02.341026 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.341235 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.341393 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:02.341487 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:02.341693 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:02.342758 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:02.342923 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:02.343086 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:02.343346 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.343559 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.343761 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:02.343885 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:02.344105 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:02.345324 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:02.345456 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:02.345620 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:02.345877 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.346127 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.346299 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:02.346398 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:02.346630 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:02.347709 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:02.347843 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:02.348014 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:02.348320 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.348569 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.348725 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:02.348817 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:02.349042 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:02.349856 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:02.350003 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:02.350177 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:02.350444 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.350664 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.350827 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:02.350921 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:02.351158 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:02.351794 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:02.352511 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:02.352917 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:02.353046 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:02.356811 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:02.358149 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.358875 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.359276 [ 31906 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:02.359523 [ 31906 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:02.548717 [ 32317 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:02.549263 [ 32317 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: ::1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:02 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE72F23653661
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:02.549687 [ 32317 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:02.640644 [ 31878 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:02.646702 [ 31878 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:02.657180 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:02.658270 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:02.684691 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.742051 [ 31872 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.742456 [ 31872 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.742515 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:02.742782 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:02.742879 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.743078 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.745186 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.745274 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.747023 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.747072 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.747327 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:02.747526 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:02.747658 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:02.747695 [ 32082 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.748093 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:02.748218 [ 32082 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.748582 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.748655 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:02.748850 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:02.748978 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:02.749070 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.749096 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:02.749812 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:02.750918 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.751733 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.752002 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:02.753345 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:02.753655 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:02.754124 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.754561 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:02.754929 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.755102 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:02.755418 [ 32103 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.755948 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:02.755956 [ 32103 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.756280 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.756328 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:02.756676 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.756824 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:02.756980 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:02.757100 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:02.757985 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.758416 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.759633 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.760009 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.761837 [ 32089 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.762197 [ 32089 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.762202 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:02.762511 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.762484 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:02.762817 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.763048 [ 31877 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:02.763152 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:02.763296 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:02.764241 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.764329 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.765419 [ 31877 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 476ms, ratio: 0.025641028, points per minute: 32.32062)
2026.05.06 08:49:02.765621 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:02.765720 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:02.765754 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:02.765766 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:02.766242 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:02.766338 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:02.766440 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:02.766522 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:02.766633 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:02.766736 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:02.766965 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:02.768188 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:02.768787 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:02.768961 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:02.769281 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:02.769687 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:02.769850 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:02.771082 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:02.771215 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:02.771321 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:02.772132 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:02.772283 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:02.776740 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:02.778019 [ 31877 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:02.778125 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:02.778237 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:02.778914 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:02.778991 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:02.779114 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:02.779256 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:02.780579 [ 31877 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:02.782422 [ 31877 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:02.783623 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:02.783727 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:02.784022 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:02.784405 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:02.785267 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:02.785507 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:02.787545 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:02.787651 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:02.787739 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:02.787898 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.788018 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.788103 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:02.788157 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:02.788268 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:02.788787 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:02.788857 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:02.788943 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:02.789085 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.789191 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.789272 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:02.789322 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:02.789426 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:02.790006 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:02.790078 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:02.790154 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:02.790287 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.790391 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.790475 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:02.790539 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:02.790637 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:02.791164 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:02.791233 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:02.791311 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:02.791442 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.791557 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.791653 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:02.791704 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:02.791801 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:02.792360 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:02.792429 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:02.792521 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:02.792653 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.792763 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.792846 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:02.792898 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:02.793008 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:02.793569 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:02.793640 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:02.793722 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:02.793861 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.793968 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.794049 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:02.794098 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:02.794198 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:02.794610 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:02.794675 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:02.794751 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:02.794883 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.794986 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:02.795069 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:02.795118 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:02.795216 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:02.795538 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:02.795946 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:02.796171 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:02.796222 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:02.799370 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:02.799997 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.800301 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.800656 [ 31860 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:02.800956 [ 31860 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:02.801234 [ 31860 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:02.801294 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:02.801627 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:02.803358 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:02.805091 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:02.805881 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:02.806022 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:02.806129 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:02.808897 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:02.811166 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:02.811304 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:02.812202 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0917
8. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003afe2f18
9. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
10. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
11. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
12. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
13. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
14. ? @ 0x00007f6e2fbe4ac3
15. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.815465 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:02.817545 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:02.817620 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:02.817738 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:02.817852 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:02.822206 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:02.822391 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:02.822890 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:02.823315 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:02.824305 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:02.824751 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:02.829454 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:02.829618 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:02.829768 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:02.830043 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.830258 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.830415 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:02.830525 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:02.830731 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:02.831688 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:02.831819 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:02.831962 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:02.832218 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.832417 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.832585 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:02.832678 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:02.832882 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:02.833938 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:02.834061 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:02.834199 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:02.834444 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.834659 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.834813 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:02.834907 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:02.835095 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:02.836087 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:02.836214 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:02.836368 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:02.836636 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.836827 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.836997 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:02.837089 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:02.837274 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:02.838372 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:02.838517 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:02.838659 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:02.838915 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.839117 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.839292 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:02.839398 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:02.839635 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:02.840613 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:02.840738 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:02.840879 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:02.841128 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.841335 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.841483 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:02.841591 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:02.841780 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:02.842524 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:02.842642 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:02.842776 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:02.843018 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.843212 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:02.843357 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:02.843466 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:02.843766 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:02.844407 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:02.845259 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:02.845778 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:02.845910 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:02.849861 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:02.851181 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.851873 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:02.852375 [ 31854 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:02.852659 [ 31854 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:02.924906 [ 31850 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1786ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:02.944846 [ 32265 ] {} <Information> KeeperDispatcher: Found dead session 4, will try to close it
2026.05.06 08:49:02.945139 [ 32265 ] {} <Information> KeeperDispatcher: Dead session close request pushed
2026.05.06 08:49:02.945262 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:49:02.945734 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:02.964775 [ 31849 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:02.970869 [ 31849 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:02.979861 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:02.980886 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:03.002306 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:03.002441 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:03.002812 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006274097 sec. (159.385 rows/sec., 3.74 KiB/sec.)
2026.05.06 08:49:03.002909 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:03.006800 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:03.007431 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:03.014165 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:03.014878 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:03.025698 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:03.028571 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:03.028946 [ 32275 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:03.128144 [ 31844 ] {} <Debug> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa) (MergerMutator): Selected 5 parts from 202605_1_166_33 to 202605_170_170_0
2026.05.06 08:49:03.128574 [ 31844 ] {} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2026.05.06 08:49:03.128820 [ 31844 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:03.129857 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Debug> MergeTask::PrepareStage: Merging 5 parts: from 202605_1_166_33 to 202605_170_170_0 into Compact with storage Full
2026.05.06 08:49:03.132020 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Debug> MergeTask::PrepareStage: Selected MergeAlgorithm: Horizontal
2026.05.06 08:49:03.132471 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_1_166_33, total 3287 rows starting from the beginning of the part
2026.05.06 08:49:03.133474 [ 32384 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:03.133879 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_167_167_0, total 14 rows starting from the beginning of the part
2026.05.06 08:49:03.134952 [ 32391 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:03.135062 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_168_168_0, total 15 rows starting from the beginning of the part
2026.05.06 08:49:03.135880 [ 32461 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:03.136230 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_169_169_0, total 3 rows starting from the beginning of the part
2026.05.06 08:49:03.137030 [ 32362 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:03.137333 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202605_170_170_0, total 8 rows starting from the beginning of the part
2026.05.06 08:49:03.138279 [ 32393 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:03.181328 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 3327 rows, containing 8 columns (8 merged, 0 gathered) in 0.051937645 sec., 64057.58289579745 rows/sec., 1.71 MiB/sec.
2026.05.06 08:49:03.186462 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Trace> MergedBlockOutputStream: filled checksums 202605_1_170_34 (state Temporary)
2026.05.06 08:49:03.191471 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Renaming temporary part tmp_merge_202605_1_170_34 to 202605_1_170_34 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:03.191769 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): preparePartForCommit: inserting 202605_1_170_34 (state PreActive) into data_parts_indexes
2026.05.06 08:49:03.191856 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Trace> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa) (MergerMutator): Merged 5 parts: [202605_1_166_33, 202605_170_170_0] -> 202605_1_170_34
2026.05.06 08:49:03.192013 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_1_166_33
2026.05.06 08:49:03.192089 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_167_167_0
2026.05.06 08:49:03.192139 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_168_168_0
2026.05.06 08:49:03.192187 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_169_169_0
2026.05.06 08:49:03.192236 [ 32271 ] {06cc28be-41de-4005-9f6a-549c04c2d0aa::202605_1_170_34} <Test> VersionMetadata: Trying to lock removal_tid by (1, 1, 00000000-0000-0000-0000-000000000000), table: system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa), part: 202605_170_170_0
2026.05.06 08:49:03.193567 [ 31839 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 4284ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:03.194821 [ 32271 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 9.16 MiB.
2026.05.06 08:49:03.227063 [ 31837 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:03.233310 [ 31837 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:03.241964 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:03.242893 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:03.262211 [ 32386 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:03.262352 [ 32386 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:03.262677 [ 32386 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006076017 sec. (164.582 rows/sec., 3.86 KiB/sec.)
2026.05.06 08:49:03.262767 [ 32386 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:03.267323 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:03.268072 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:03.274849 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:03.275547 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:03.287732 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:03.291152 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:03.291735 [ 32285 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:03.516329 [ 31804 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:03.516772 [ 31804 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:03.516830 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:03.517106 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:03.517235 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:03.517391 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:03.519639 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:03.519800 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:03.521546 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:03.521592 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:03.521777 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:03.521912 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:03.522016 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:03.522076 [ 31801 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:03.522512 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:03.522594 [ 31801 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:03.522799 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:03.522860 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:03.522892 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:03.522911 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:03.523052 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:03.523185 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:03.523694 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:03.524708 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:03.525270 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:03.525368 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:03.526134 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:03.526319 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:03.526468 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:03.526650 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:03.526898 [ 31796 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:03.527113 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:03.527353 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:03.527785 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:03.528127 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:03.528604 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:03.528554 [ 31794 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:03.528748 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:03.528883 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:03.528967 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:03.529120 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:03.529679 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:03.530561 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:03.531392 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:03.532061 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:03.532343 [ 31791 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:03.532670 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:03.532725 [ 31791 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:03.532996 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:03.533017 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:03.533279 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:03.534259 [ 31802 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:03.534379 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:03.534520 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:03.534986 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:03.534991 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:03.536725 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:03.536872 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:03.536964 [ 31802 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 755ms, ratio: 0.025641028, points per minute: 20.376976)
2026.05.06 08:49:03.537125 [ 31790 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:03.537184 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:03.537320 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:03.537452 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:03.537603 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:03.537792 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:03.537900 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:03.537913 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:03.538001 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:03.538017 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:03.538123 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:03.538701 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:03.539073 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:03.540049 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:03.540545 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:03.540691 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:03.540773 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:03.540859 [ 31782 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:03.540959 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:03.541220 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:03.541271 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:03.541467 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:03.541804 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:03.543104 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:03.543242 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:03.543353 [ 31802 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:03.543362 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:03.543563 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:03.543696 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:03.545477 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:03.547732 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:03.549248 [ 31802 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:03.549355 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:03.549486 [ 31802 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:03.549953 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:03.550028 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:03.550149 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:03.550263 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:03.552175 [ 31802 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:03.554307 [ 31802 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:03.555011 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:03.555218 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:03.555671 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:03.556215 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:03.557186 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:03.557583 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:03.562476 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:03.562640 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:03.562802 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:03.563076 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.563285 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.563442 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:03.563553 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:03.563786 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:03.564728 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:03.564853 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:03.564993 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:03.565264 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.565528 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.565696 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:03.565789 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:03.565987 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:03.567102 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:03.567228 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:03.567366 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:03.567672 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.567869 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.568021 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:03.568113 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:03.568300 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:03.569276 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:03.569397 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:03.569546 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:03.569786 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.570005 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.570181 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:03.570274 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:03.570478 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:03.571575 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:03.571734 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:03.571874 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:03.572127 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.572326 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.572479 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:03.572586 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:03.572780 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:03.573740 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:03.573859 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:03.573999 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:03.574247 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.574440 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.574617 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:03.574705 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:03.574894 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:03.575742 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:03.575869 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:03.576003 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:03.576280 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.576476 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:03.576642 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:03.576731 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:03.576922 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:03.577450 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:03.578146 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:03.578565 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:03.578654 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:03.582490 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:03.584055 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:03.584758 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:03.585248 [ 31773 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:03.585652 [ 31773 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:03.585992 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:03.586008 [ 31773 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:03.586331 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:03.588242 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:03.590117 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:03.590904 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:03.591009 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:03.591105 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:03.593865 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:03.596175 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:03.596315 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:03.600718 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:03.604632 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:03.604719 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:03.604855 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:03.604983 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:03.609831 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:03.610014 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:03.610582 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:03.611064 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:03.612291 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:03.612676 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:03.617167 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:03.617313 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:03.617472 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:03.617762 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.617974 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.618128 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:03.618224 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:03.618426 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:03.619335 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:03.619460 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:03.619671 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:03.619929 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.620129 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.620282 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:03.620374 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:03.620577 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:03.621637 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:03.621763 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:03.621900 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:03.622145 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.622337 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.622487 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:03.622593 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:03.622779 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:03.623785 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:03.623912 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:03.624046 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:03.624287 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.624476 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.624643 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:03.624736 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:03.624933 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:03.625984 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:03.626107 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:03.626246 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:03.626509 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.626709 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.626863 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:03.626956 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:03.627145 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:03.628154 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:03.628280 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:03.628420 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:03.628682 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.628877 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.629046 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:03.629135 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:03.629323 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:03.630067 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:03.630183 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:03.630313 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:03.630567 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.630759 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:03.630904 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:03.630994 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:03.631182 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:03.631791 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:03.632434 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:03.632842 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:03.632932 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:03.636621 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:03.637932 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:03.638562 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:03.638968 [ 31768 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:03.639162 [ 31768 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:03.718148 [ 31767 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:03.724357 [ 31767 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:03.730485 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:03.731236 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:03.749064 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:03.749575 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:03.771800 [ 31758 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 7566ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:04.019278 [ 31748 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1094ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:04.665741 [ 31718 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:04.666158 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:04.666171 [ 31718 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:04.666591 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:04.666632 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:04.666934 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:04.668698 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:04.668744 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:04.670302 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:04.670363 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:04.670607 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:04.670848 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:04.670964 [ 31716 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:04.670966 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:04.671317 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:04.671363 [ 31716 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:04.671584 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:04.671636 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:04.671755 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:04.672055 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:04.672177 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:04.672302 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:04.672991 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:04.673483 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:04.673585 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:04.674540 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:04.675069 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:04.675090 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:04.675250 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:04.675444 [ 31714 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:04.675466 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:04.675764 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:04.675825 [ 31714 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:04.675913 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:04.676067 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:04.676234 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:04.676394 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:04.676657 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:04.677780 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:04.677928 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:04.678044 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:04.678479 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:04.678689 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:04.680323 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:04.680625 [ 31710 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:04.680866 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:04.680936 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:04.681196 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:04.682861 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:04.683253 [ 31715 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:04.683357 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:04.683524 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:04.684558 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:04.685310 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:04.685419 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:04.685534 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:04.685961 [ 31715 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1132ms, ratio: 0.025641028, points per minute: 13.59065)
2026.05.06 08:49:04.686154 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:04.686250 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:04.686863 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:04.686957 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:04.687055 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:04.687561 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:04.688261 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:04.688897 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:04.689639 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:04.689829 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:04.690130 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:04.690525 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:04.690545 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:04.690665 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:04.691764 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:04.691907 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:04.692018 [ 31715 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:04.694624 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:04.696874 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:04.696950 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:04.697070 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:04.697183 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:04.698004 [ 31715 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:04.698131 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:04.698284 [ 31715 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:04.699566 [ 32318 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Unloaded primary key for outdated part 202605_169_169_0
2026.05.06 08:49:04.699709 [ 32318 ] {} <Test> system.error_log (06cc28be-41de-4005-9f6a-549c04c2d0aa): Unloaded primary key for outdated part 202605_170_170_0
2026.05.06 08:49:04.702106 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:04.702216 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:04.702794 [ 31715 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:49:04.702893 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:04.703469 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:04.704590 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:04.704893 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:04.707289 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:04.707391 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:04.707488 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:04.707731 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.707878 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.707981 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:04.708045 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:04.708170 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:04.708755 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:04.708829 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:04.708911 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:04.709070 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.709194 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.709282 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:04.709335 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:04.709446 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:04.710058 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:04.710131 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:04.710210 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:04.710342 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.710447 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.710546 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:04.710599 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:04.710698 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:04.711225 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:04.711292 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:04.711369 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:04.711516 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.711634 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.711718 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:04.711768 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:04.711862 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:04.712423 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:04.712506 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:04.712587 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:04.712721 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.712827 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.712910 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:04.712960 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:04.713069 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:04.713614 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:04.713689 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:04.713769 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:04.713912 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.714029 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.714124 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:04.714176 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:04.714281 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:04.714717 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:04.714787 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:04.714870 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:04.715004 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.715110 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:04.715190 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:04.715239 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:04.715339 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:04.715716 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:04.716168 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:04.716427 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:04.716490 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:04.718868 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:04.719476 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:04.719849 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:04.720125 [ 31723 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:04.720316 [ 31723 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:04.720319 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:04.720527 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:04.722062 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:04.723265 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:04.724032 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:04.724103 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:04.724164 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:04.725878 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:04.727071 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:04.727201 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:04.729523 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:04.730540 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:04.730580 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:04.730660 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:04.730755 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:04.734370 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:04.734546 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:04.735014 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:04.735646 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:04.736534 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:04.736865 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:04.740722 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:04.740851 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:04.740980 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:04.741216 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.741435 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.741596 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:04.741680 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:04.741854 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:04.742687 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:04.742799 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:04.742923 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:04.743139 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.743330 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.743521 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:04.743662 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:04.743867 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:04.744792 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:04.744938 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:04.745064 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:04.745281 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.745445 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.745591 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:04.745680 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:04.745859 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:04.746723 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:04.746865 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:04.746996 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:04.747209 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.747379 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.747522 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:04.747602 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:04.747792 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:04.748729 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:04.748869 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:04.749004 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:04.749225 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.749393 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.749538 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:04.749617 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:04.749784 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:04.750644 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:04.750751 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:04.750907 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:04.751147 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.751312 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.751436 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:04.751526 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:04.751711 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:04.752379 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:04.752483 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:04.752615 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:04.752835 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.753031 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:04.753196 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:04.753281 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:04.753510 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:04.753966 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:04.754548 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:04.754882 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:04.754955 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:04.757537 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:04.758632 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:04.759160 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:04.759512 [ 31700 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:04.827068 [ 31697 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:04.831135 [ 31697 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:04.832514 [ 31693 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 10617ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:04.840665 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:04.841825 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:04.866302 [ 32437 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:04.866418 [ 32437 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:04.866647 [ 32437 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007160669 sec. (139.652 rows/sec., 3.27 KiB/sec.)
2026.05.06 08:49:04.866714 [ 32437 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:04.872258 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:04.873103 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:04.880245 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:04.880856 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:04.893158 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:04.896366 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:04.896866 [ 32282 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:05.067119 [ 31709 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.067444 [ 31709 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.067661 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.067845 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.068061 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.068240 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.070344 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.070391 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.072249 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.072306 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.072546 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:05.072712 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:05.072838 [ 31900 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.072858 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:05.073238 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.073295 [ 31900 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.073652 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.073663 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.073716 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:05.073864 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:05.073922 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.073990 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:05.074679 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:05.075402 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.075436 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.076094 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:05.076824 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:05.076966 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.076998 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.077043 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:05.077357 [ 31865 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.077418 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:05.077555 [ 31865 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.077628 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.077769 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.077881 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.077905 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.078060 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.079081 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:05.079274 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:05.079398 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:05.079696 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.079811 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.081596 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.081645 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.082112 [ 32078 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.082577 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.082602 [ 32078 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.082906 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.082969 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.083174 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.085031 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.085171 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.085260 [ 31899 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:05.085369 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:05.085665 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:05.086941 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:05.087013 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.087397 [ 31866 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.087865 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.087933 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:05.088048 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:05.088139 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.088151 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:05.088899 [ 31899 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 387ms, ratio: 0.025641028, points per minute: 39.75353)
2026.05.06 08:49:05.089100 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:05.089200 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:05.089910 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:05.090018 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:05.090122 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:05.090275 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.090689 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:05.091151 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:05.092067 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.092150 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:05.092553 [ 32068 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.092909 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.092970 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:05.093164 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:05.093231 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.093516 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:05.093583 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:05.093736 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:05.094012 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.095419 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:05.095585 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:05.095764 [ 31899 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:05.096126 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.096378 [ 31640 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #5
2026.05.06 08:49:05.096423 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:49:05.096710 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.097740 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:05.098856 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.099274 [ 32067 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.099706 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.100014 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.100186 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:05.100262 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:05.100396 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:05.100533 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:05.101641 [ 31899 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:05.101755 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:05.101924 [ 31899 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:05.101939 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.103953 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:05.104612 [ 31899 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:05.104908 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:05.105024 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:05.105129 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:05.105216 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:05.105393 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:05.105769 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:05.106299 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.106927 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:05.107225 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:05.107638 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:05.107964 [ 31899 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 4ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:05.108246 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:05.108368 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:05.110627 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:05.112237 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:05.112383 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:05.112555 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:05.112836 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.113048 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.113215 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:05.113317 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:05.113535 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:05.114071 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:05.114167 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:05.114290 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:05.114436 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:05.114443 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:05.114660 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:05.114806 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:05.115060 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.115259 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.115430 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:05.115537 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:05.115765 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:05.116829 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:05.116953 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:05.117091 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:05.117359 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.117571 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.117725 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:05.117819 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:05.118005 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:05.119001 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:05.119124 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:05.119261 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:05.119524 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.119765 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.119934 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:05.120026 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:05.120201 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:05.120211 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:05.120459 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:05.120981 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:05.121361 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:05.121489 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:05.121541 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.121654 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:05.121907 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.122118 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.122272 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:05.122366 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:05.122572 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:05.122577 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:05.123095 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:05.123574 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:05.123730 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:05.123873 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:05.124123 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.124318 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.124468 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:05.124573 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:05.124763 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:05.125508 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:05.125627 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:05.125761 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:05.126006 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.126202 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:05.126360 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:05.126449 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:05.126654 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:05.127163 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:05.127898 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:05.128296 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:05.128384 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:05.128916 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:05.129078 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:05.129244 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:05.129550 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.129768 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.129929 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:05.130064 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:05.130274 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:05.131249 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:05.131379 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:05.131542 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:05.131629 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:05.131848 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.132055 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.132214 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:05.132310 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:05.132524 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:05.132951 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.133628 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:05.133602 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.133760 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:05.133905 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:05.134162 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.134361 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.134534 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:05.134629 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:05.134822 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:05.135873 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:05.136005 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:05.136166 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:05.136421 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.136636 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.136794 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:05.136890 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:05.137084 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:05.138177 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:05.138305 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:05.138450 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:05.138725 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.138934 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.139091 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:05.139188 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:05.139386 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:05.140437 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:05.140585 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:05.140728 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:05.140981 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.141179 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.141330 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:05.141421 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:05.141632 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:05.142390 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:05.142530 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:05.142669 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:05.142922 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.143120 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.143269 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:05.143361 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:05.143586 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:05.144176 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:05.144886 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:05.145304 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:05.145397 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:05.149392 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:05.150728 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.151374 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.151882 [ 32072 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:05.152123 [ 32072 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:05.155962 [ 32061 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:05.161725 [ 32061 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:05.171098 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:05.172183 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:05.194352 [ 32319 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:05.194567 [ 32319 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:05.195509 [ 32319 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007284829 sec. (137.272 rows/sec., 3.22 KiB/sec.)
2026.05.06 08:49:05.195657 [ 32319 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:05.199805 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:05.200467 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:05.207535 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:05.208258 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:05.221008 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.225036 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.225641 [ 32276 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:05.418666 [ 32073 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:05.424458 [ 32073 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:05.430828 [ 32104 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9533ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:05.434010 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:05.435085 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:05.457626 [ 32327 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:05.457773 [ 32327 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:05.458076 [ 32327 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006760238 sec. (147.924 rows/sec., 3.47 KiB/sec.)
2026.05.06 08:49:05.458163 [ 32327 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:05.462204 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:05.462877 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:05.469901 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:05.470573 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:05.482834 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.485951 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.486423 [ 32273 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:05.571546 [ 31862 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1552ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:05.638753 [ 32171 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.639169 [ 32171 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.639206 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.639541 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.639666 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.639977 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.642354 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.642387 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.644304 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.644339 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.644579 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:05.644771 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:05.644876 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:05.644960 [ 32155 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.645264 [ 32155 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.645353 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.645457 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.645519 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:05.645620 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:05.645657 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.645742 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:05.645748 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.646340 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:05.647405 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.647506 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.647966 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:05.648802 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:05.649063 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:05.649217 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.649349 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.649460 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:05.649610 [ 32105 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.649952 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.650067 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.650119 [ 32105 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.650381 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.650380 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.650703 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.651318 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:05.651455 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:05.651594 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:05.652417 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.652441 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.654146 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.654461 [ 32166 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.654483 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.654855 [ 32166 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.654864 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.655142 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.655139 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.655439 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.657011 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.657166 [ 32126 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:05.657267 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.657268 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:05.657509 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:05.658693 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.659016 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:05.659065 [ 32083 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.659392 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.659677 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.659907 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:05.659963 [ 32126 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 552ms, ratio: 0.025641028, points per minute: 27.870682)
2026.05.06 08:49:05.660027 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:05.660130 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:05.660155 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:05.660256 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:05.660830 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:05.660928 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:05.661046 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:05.661175 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.661604 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:05.662736 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.662821 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:05.662919 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:05.663120 [ 32146 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.663534 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.663858 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:05.663871 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.664055 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:05.664337 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:05.664798 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.665125 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:05.665270 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:05.666018 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:05.666123 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.666158 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:05.666271 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:05.667913 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.669211 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:05.671588 [ 32126 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:05.671637 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:05.671712 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:05.671717 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:05.671830 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:05.671842 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:05.671941 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:05.674299 [ 32126 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:05.676587 [ 32126 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:05.676711 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:05.676885 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:05.677315 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:05.677883 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.678769 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:05.679117 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:05.683598 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:05.683772 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:05.683921 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:05.684206 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.684426 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.684617 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:05.684715 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:05.684991 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:05.685948 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:05.686090 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:05.686238 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:05.686518 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.686728 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.686888 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:05.686981 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:05.687177 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:05.688271 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:05.688400 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:05.688554 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:05.688809 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.689026 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.689186 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:05.689281 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:05.689475 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:05.690458 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:05.690599 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:05.690739 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:05.690997 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.691197 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.691351 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:05.691445 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:05.691696 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:05.692761 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:05.692922 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:05.693070 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:05.693328 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.693556 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.693713 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:05.693808 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:05.694007 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:05.694959 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:05.695082 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:05.695224 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:05.695481 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.695740 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.695896 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:05.695989 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:05.696194 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:05.696956 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:05.697075 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:05.697211 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:05.697465 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.697681 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:05.697828 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:05.697918 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:05.698111 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:05.698640 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:05.699283 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:05.699744 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:05.699838 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:05.704026 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:05.705321 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.705971 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.706320 [ 32150 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.706644 [ 32150 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:05.706712 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.707004 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.708789 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.710557 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:05.711330 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:05.711430 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:05.711540 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:05.714303 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:05.716597 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:05.716736 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:05.720876 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:05.722902 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:05.722971 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:05.723084 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:05.723191 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:05.727859 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:05.728034 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:05.728517 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:05.729059 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.730179 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:05.730559 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:05.735051 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:05.735188 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:05.735327 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:05.735643 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.735854 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.736023 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:05.736115 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:05.736310 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:05.737189 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:05.737312 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:05.737448 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:05.737717 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.737908 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.738057 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:05.738146 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:05.738329 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:05.739353 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:05.739475 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:05.739663 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:05.739913 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.740116 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.740266 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:05.740357 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:05.740553 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:05.741652 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:05.741786 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:05.741926 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:05.742180 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.742373 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.742541 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:05.742637 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:05.742844 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:05.743963 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:05.744095 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:05.744236 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:05.744525 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.744730 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.744883 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:05.744975 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:05.745165 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:05.746134 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:05.746255 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:05.746392 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:05.746659 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.746853 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.746999 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:05.747086 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:05.747272 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:05.748042 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:05.748172 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:05.748304 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:05.748561 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.748751 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.748892 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:05.748979 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:05.749162 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:05.749668 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:05.750301 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:05.750701 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:05.750788 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:05.754966 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:05.756291 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.756937 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.757330 [ 32098 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:05.899042 [ 32141 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.899545 [ 32141 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.899668 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.899895 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.900102 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.900200 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.902431 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.902477 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.904170 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.904263 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.904451 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:05.904635 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:05.904736 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:05.904810 [ 32182 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.905151 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.905267 [ 32182 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.905480 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:05.905595 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:05.905604 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.905653 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.905710 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:05.905945 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.906319 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:05.907435 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.907773 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:05.907805 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.908464 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:05.908690 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:05.909078 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.909106 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:05.909463 [ 31880 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.909606 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.909816 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.910113 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.910646 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.911003 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:05.911030 [ 32162 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.911168 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:05.911334 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:05.911341 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.911736 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.912022 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.913327 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.913698 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.914190 [ 32151 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.914561 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:05.914837 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.915482 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:05.916795 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.917358 [ 32080 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:05.917466 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:05.917634 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:05.918514 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:05.919292 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:05.919399 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:05.919510 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:05.920220 [ 32080 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 244ms, ratio: 0.025641028, points per minute: 63.05171)
2026.05.06 08:49:05.920447 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:05.920590 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:05.921224 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:05.921341 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:05.921462 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:05.922097 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:05.922713 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:05.923790 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:05.924615 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:05.924846 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:05.925205 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:05.925252 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:05.925394 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:05.925658 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.927042 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:05.927221 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:05.927360 [ 32080 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:05.930127 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:05.932337 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:05.932413 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:05.932546 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:05.932659 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:05.933685 [ 32080 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:05.933786 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:05.933905 [ 32080 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:05.936814 [ 32080 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:49:05.937293 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:05.937470 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:05.937922 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:05.938430 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.939398 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:05.939820 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:05.944349 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:05.944491 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:05.944659 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:05.944934 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.945159 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.945317 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:05.945414 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:05.945631 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:05.946558 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:05.946686 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:05.946824 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:05.947077 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.947273 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.947425 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:05.947534 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:05.947793 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:05.948887 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:05.949029 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:05.949190 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:05.949448 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.949662 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.949819 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:05.949916 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:05.950107 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:05.951096 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:05.951221 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:05.951358 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:05.951654 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.951852 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.952005 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:05.952100 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:05.952286 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:05.953359 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:05.953485 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:05.953649 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:05.953904 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.954109 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.954266 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:05.954373 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:05.954582 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:05.955555 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:05.955714 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:05.955860 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:05.956113 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.956309 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.956459 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:05.956562 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:05.956766 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:05.957519 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:05.957640 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:05.957798 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:05.958062 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.958257 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:05.958404 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:05.958509 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:05.958699 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:05.959221 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:05.959921 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:05.960330 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:05.960421 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:05.964082 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:05.965398 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.966056 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:05.966643 [ 32173 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:05.967115 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:05.967084 [ 32173 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:05.967451 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:05.969457 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:05.971194 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:05.971993 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:05.972102 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:05.972199 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:05.974917 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:05.977231 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:05.977376 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:05.981335 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:05.983507 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:05.983579 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:05.983730 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:05.983843 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:05.988438 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:05.988646 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:05.989142 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:05.989795 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:05.990855 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:05.991247 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:05.995801 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:05.995949 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:05.996100 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:05.996381 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.996629 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.996790 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:05.996888 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:05.997094 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:05.998042 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:05.998170 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:05.998313 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:05.998582 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.998794 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:05.998946 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:05.999038 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:05.999229 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:06.000344 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:06.000475 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:06.000634 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:06.000901 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.001099 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.001255 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:06.001351 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:06.001560 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:06.002540 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:06.002666 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:06.002806 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:06.003056 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.003253 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.003407 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:06.003518 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:06.003747 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:06.004846 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:06.004987 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:06.005131 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:06.005396 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.005622 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.005779 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:06.005877 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:06.006076 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:06.007060 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:06.007185 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:06.007329 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:06.007596 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.007841 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.007994 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:06.008084 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:06.008277 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:06.009052 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:06.009173 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:06.009310 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:06.009578 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.009774 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.009920 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:06.010009 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:06.010201 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:06.010742 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:06.011440 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:06.011888 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:06.011983 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:06.015898 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:06.017236 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:06.017894 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:06.018348 [ 32147 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:06.018560 [ 32147 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:06.180598 [ 32138 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:06.183184 [ 32140 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 5269ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:06.186416 [ 32138 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:06.192425 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:06.193169 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:06.208445 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:06.208617 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:06.208913 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.005151007 sec. (194.137 rows/sec., 4.55 KiB/sec.)
2026.05.06 08:49:06.209049 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:06.212027 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:06.212456 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:06.216722 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:06.217215 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:06.224828 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:06.227204 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:06.227596 [ 32271 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:06.322931 [ 32175 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 4389ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:06.605139 [ 32062 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:06.605570 [ 32062 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:06.605615 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:06.605867 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:06.605950 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:06.606150 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:06.607942 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:06.608059 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:06.609479 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:06.609555 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:06.609660 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:06.609732 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:06.609788 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:06.610180 [ 32054 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:06.610405 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:06.610459 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:06.610529 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:06.610649 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:06.610658 [ 32054 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:06.610848 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:06.611011 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:06.611019 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:06.611339 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:06.612158 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:06.612703 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:06.612773 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:06.612886 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:06.613110 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:06.613217 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:06.613734 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:06.615004 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:06.615078 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:06.615135 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:06.615211 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:06.615708 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:06.615714 [ 32045 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:06.616048 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:06.616111 [ 32045 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:06.616370 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:06.616394 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:06.616685 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:06.617475 [ 32059 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:06.617541 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:06.617611 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:06.618084 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:06.618347 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:06.619538 [ 32059 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 683ms, ratio: 0.025641028, points per minute: 22.52506)
2026.05.06 08:49:06.619658 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:06.619704 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:06.619724 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:06.620028 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:06.620321 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:06.620383 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:06.620445 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:06.620621 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:06.620740 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:06.620760 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:06.620845 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:06.621993 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:06.622848 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:06.622958 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:06.623295 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:06.623784 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:06.624266 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:06.625021 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:06.625097 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:06.625154 [ 32059 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:06.627029 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:06.627207 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:06.627966 [ 32059 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:06.628031 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:06.628113 [ 32059 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:06.630266 [ 32059 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 11ms, ratio: 0.025641028, points per minute: 1398.6014)
2026.05.06 08:49:06.632130 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:06.632215 [ 32059 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:06.634057 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:06.634131 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:06.634253 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:06.634367 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:06.639363 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:06.639570 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:06.640056 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:06.640746 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:06.641755 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:06.642174 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:06.646871 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:06.647023 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:06.647179 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:06.647464 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.647744 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.647912 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:06.648012 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:06.648224 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:06.649181 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:06.649307 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:06.649449 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:06.649728 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.649934 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.650092 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:06.650200 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:06.650393 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:06.651473 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:06.651653 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:06.651801 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:06.652059 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.652260 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.652419 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:06.652530 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:06.652723 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:06.653710 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:06.653836 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:06.653979 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:06.654235 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.654466 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.654693 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:06.654794 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:06.655006 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:06.656324 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:06.656462 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:06.656630 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:06.656899 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.657109 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.657269 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:06.657366 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:06.657588 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:06.658668 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:06.658800 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:06.658950 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:06.659209 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.659427 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.659600 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:06.659730 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:06.659927 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:06.660710 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:06.660831 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:06.660969 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:06.661271 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.661474 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:06.661641 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:06.661734 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:06.661929 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:06.662475 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:06.663245 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:06.663782 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:06.663883 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:06.667998 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:06.669336 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:06.670003 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:06.670419 [ 32051 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:06.670688 [ 32051 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:06.671024 [ 32051 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:06.671064 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:06.671389 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:06.673298 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:06.674944 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:06.675747 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:06.675861 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:06.675964 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:06.678903 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:06.681163 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:06.681314 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:06.685483 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:06.687954 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:06.688041 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:06.688168 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:06.688286 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:06.692478 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:06.692685 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:06.693164 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:06.693725 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:06.694724 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:06.695122 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:06.699684 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:06.699838 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:06.699994 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:06.700278 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.700515 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.700680 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:06.700778 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:06.700984 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:06.701929 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:06.702060 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:06.702204 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:06.702463 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.702687 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.702846 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:06.702956 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:06.703155 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:06.704287 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:06.704419 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:06.704580 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:06.704833 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.705032 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.705190 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:06.705297 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:06.705491 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:06.706478 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:06.706626 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:06.706784 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:06.707236 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.707449 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.707682 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:06.707783 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:06.707978 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:06.709085 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:06.709215 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:06.709362 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:06.709642 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.709855 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.710011 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:06.710109 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:06.710308 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:06.711302 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:06.711433 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:06.711597 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:06.711916 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.712140 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.712297 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:06.712389 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:06.712599 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:06.713362 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:06.713486 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:06.713644 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:06.713895 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.714095 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:06.714244 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:06.714337 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:06.714546 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:06.715087 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:06.715829 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:06.716257 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:06.716350 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:06.719995 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:06.721311 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:06.721971 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:06.722379 [ 32037 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:06.722595 [ 32037 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:07.078890 [ 32008 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1508ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:07.193558 [ 32011 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:07.196843 [ 32011 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:07.208325 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:07.209688 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:07.236221 [ 32349 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:07.236365 [ 32349 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:07.236697 [ 32349 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007531729 sec. (132.772 rows/sec., 3.11 KiB/sec.)
2026.05.06 08:49:07.236786 [ 32349 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:07.243451 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:07.244179 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:07.251748 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:07.252565 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:07.265180 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:07.267656 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:07.267958 [ 32271 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:07.353412 [ 32001 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 2521ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:07.424747 [ 32193 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1994ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:07.532140 [ 31991 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:07.532646 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:07.532654 [ 31991 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:07.532997 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:07.533024 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:07.533314 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:07.535310 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:07.535361 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:07.536903 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:07.536946 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:07.537138 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:07.537246 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:07.537310 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:07.537383 [ 31982 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:07.537679 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:07.537761 [ 31982 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:07.537976 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:07.538043 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:07.538177 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:07.538233 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:07.538297 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:07.538339 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:07.538656 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:07.539789 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:07.539913 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:07.540038 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:07.540532 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:07.540626 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:07.541002 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:07.541466 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:07.542486 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:07.542521 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:07.542789 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:07.542903 [ 31985 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:07.542950 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:07.543070 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:07.543205 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:07.543317 [ 31985 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:07.543472 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:07.543586 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:07.543876 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:07.545152 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:07.545634 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:07.546704 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:07.546958 [ 31978 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:07.547364 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:07.547410 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:07.547705 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:07.549406 [ 31986 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:07.549469 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:07.549529 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:07.549693 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:07.551151 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:07.551560 [ 31977 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:07.552124 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:07.552191 [ 31977 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:07.552420 [ 31986 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 920ms, ratio: 0.025641028, points per minute: 16.722408)
2026.05.06 08:49:07.552521 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:07.552614 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:07.552650 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:07.552871 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:07.553126 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:07.554051 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:07.554169 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:07.554277 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:07.554857 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:07.556491 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:07.556546 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:07.557186 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:07.557903 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:07.558108 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:07.558953 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:07.559401 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:07.559580 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:07.559565 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:07.560528 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:07.560596 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:07.560661 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:07.561028 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:07.561169 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:07.561280 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:07.563470 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:07.565780 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:07.565925 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:07.566401 [ 31986 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:07.566518 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:07.566640 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:07.569345 [ 31986 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:49:07.569980 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:07.571872 [ 31986 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:07.572678 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:07.572777 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:07.572932 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:07.573086 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:07.577948 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:07.578139 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:07.578609 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:07.579220 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:07.580146 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:07.580629 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:07.585947 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:07.586101 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:07.586256 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:07.586594 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.586834 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.587001 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:07.587103 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:07.587311 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:07.588372 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:07.588559 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:07.588730 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:07.589003 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.589207 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.589379 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:07.589479 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:07.589745 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:07.591022 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:07.591158 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:07.591301 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:07.591681 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.591906 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.592069 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:07.592168 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:07.592359 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:07.593421 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:07.593621 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:07.593775 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:07.594057 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.594316 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.594478 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:07.594625 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:07.594843 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:07.596054 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:07.596190 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:07.596340 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:07.596634 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.596881 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.597045 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:07.597142 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:07.597340 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:07.598488 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:07.598685 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:07.598864 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:07.599132 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.599331 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.599483 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:07.599710 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:07.599919 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:07.600759 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:07.600893 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:07.601036 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:07.601298 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.601523 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:07.601727 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:07.601831 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:07.602030 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:07.602640 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:07.603334 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:07.603865 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:07.603971 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:07.607442 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:07.608579 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:07.609108 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:07.609490 [ 31962 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:07.609791 [ 31962 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:07.610078 [ 31962 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:07.610162 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:07.610556 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:07.612592 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:07.614548 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:07.615541 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:07.615689 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:07.615796 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:07.618968 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:07.621337 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:07.621547 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:07.625772 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:07.628188 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:07.628269 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:07.628391 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:07.628521 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:07.633394 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:07.633599 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:07.634127 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:07.634783 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:07.635921 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:07.636312 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:07.641357 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:07.641521 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:07.641672 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:07.641953 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.642168 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.642328 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:07.642423 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:07.642660 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:07.643586 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:07.643750 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:07.643892 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:07.644148 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.644354 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.644570 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:07.644676 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:07.644874 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:07.645946 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:07.646073 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:07.646211 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:07.646457 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.646670 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.646826 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:07.646921 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:07.647110 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:07.648108 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:07.648236 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:07.648374 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:07.648637 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.648856 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.649009 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:07.649103 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:07.649288 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:07.650345 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:07.650469 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:07.650629 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:07.650879 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.651082 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.651234 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:07.651328 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:07.651534 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:07.652553 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:07.652679 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:07.652834 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:07.653095 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.653291 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.653439 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:07.653539 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:07.653727 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:07.654612 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:07.654756 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:07.654899 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:07.655156 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.655354 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:07.655520 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:07.655643 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:07.655838 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:07.656373 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:07.657037 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:07.657430 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:07.657534 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:07.661202 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:07.662543 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:07.663182 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:07.663564 [ 31960 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:07.663770 [ 31960 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:08.016230 [ 31947 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:08.022388 [ 31947 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:08.030231 [ 31948 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:08.030581 [ 31948 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:08.030672 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:08.030902 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:08.031021 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:08.031270 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:08.032205 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:08.033314 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:08.033647 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:08.033758 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:08.035978 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:08.036076 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:08.036203 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:08.036399 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:08.036520 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:08.036538 [ 31945 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:08.036988 [ 31945 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:08.037065 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:08.037301 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:08.037487 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:08.037531 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:08.037620 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:08.037652 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:08.037743 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:08.038367 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:08.039890 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:08.039974 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:08.040053 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:08.040826 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:08.041041 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:08.041459 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:08.041846 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:08.041959 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:08.042872 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:08.043307 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:08.043326 [ 31940 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:08.043516 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:08.043729 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:08.045659 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:08.045731 [ 31940 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:08.046038 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:08.046060 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:08.046408 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:08.048160 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:08.048440 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:08.049655 [ 31943 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:08.049757 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:08.049909 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:08.050111 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:08.050244 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:08.051040 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:08.051149 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:08.051251 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:08.052591 [ 31943 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 480ms, ratio: 0.025641028, points per minute: 32.051285)
2026.05.06 08:49:08.052786 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:08.052884 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:08.053522 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:08.053618 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:08.053719 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:08.053973 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:08.054234 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:08.055948 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:08.056311 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:08.056455 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:08.056835 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:08.057034 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:08.057373 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:08.057818 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:08.059070 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:08.059268 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:08.059421 [ 31943 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:08.060235 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=7, jobs=7). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:08.060457 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:08.060803 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=7, jobs=7). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:08.062587 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:08.062662 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:08.062783 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:08.062897 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:08.065665 [ 31943 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:08.065774 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:08.065902 [ 31943 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:08.067577 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:08.067813 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:08.068215 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:08.068772 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:08.069562 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:08.069786 [ 31943 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:49:08.069957 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:08.074727 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:08.074879 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:08.075035 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:08.075321 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.075555 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.075761 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:08.075863 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:08.076073 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:08.077033 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:08.077163 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:08.077310 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:08.077588 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.077791 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.077948 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:08.078042 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:08.078254 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:08.079360 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:08.079487 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:08.079678 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:08.079935 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.080132 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.080291 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:08.080386 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:08.080593 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:08.081587 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:08.081714 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:08.081854 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:08.082107 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.082303 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.082471 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:08.082582 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:08.082772 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:08.083906 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:08.084039 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:08.084185 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:08.084440 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.084664 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.084822 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:08.084919 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:08.085117 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:08.086106 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:08.086231 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:08.086375 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:08.086643 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.086855 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.087008 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:08.087100 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:08.087293 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:08.088115 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:08.088241 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:08.088380 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:08.088650 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.088848 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:08.088998 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:08.089089 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:08.089280 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:08.089852 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:08.090567 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:08.091010 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:08.091117 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:08.094830 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:08.096343 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:08.097050 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:08.097509 [ 31931 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:08.097827 [ 31931 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:08.098184 [ 31931 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:08.098230 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:08.098581 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:08.100622 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:08.102669 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:08.103595 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:08.103792 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:08.103913 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:08.107675 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:08.110738 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:08.110913 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:08.114975 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:08.117673 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:08.117760 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:08.117895 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:08.118014 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:08.122832 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:08.123023 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:08.123711 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:08.124334 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:08.125401 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:08.125840 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:08.130428 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:08.130596 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:08.130754 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:08.131044 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.131270 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.131446 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:08.131589 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:08.131868 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:08.132905 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:08.133040 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:08.133190 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:08.133460 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.133693 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.133855 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:08.133952 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:08.134168 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:08.135333 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:08.135476 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:08.135714 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:08.135996 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.136226 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.136395 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:08.136509 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:08.136713 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:08.137719 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:08.137849 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:08.137994 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:08.138258 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.138464 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.138658 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:08.138754 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:08.138952 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:08.140083 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:08.140216 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:08.140366 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:08.140650 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.140866 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.141036 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:08.141136 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:08.141340 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:08.142325 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:08.142455 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:08.142622 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:08.142889 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.143114 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.143274 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:08.143367 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:08.143581 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:08.144440 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:08.144599 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:08.144747 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:08.145015 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.145232 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:08.145388 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:08.145482 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:08.145717 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:08.146377 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:08.147114 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:08.147556 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:08.147695 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:08.152058 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:08.153421 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:08.154105 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:08.154538 [ 31926 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:08.363418 [ 32318 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:08.363989 [ 32318 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: ::1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:08 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE7307DFBAC53
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:08.364404 [ 32318 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:08.422523 [ 31980 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1343ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:08.621421 [ 31893 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:08.627748 [ 31893 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:08.636930 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:08.637993 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:08.660078 [ 32432 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:08.660226 [ 32432 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:08.660601 [ 32432 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006457888 sec. (154.849 rows/sec., 3.63 KiB/sec.)
2026.05.06 08:49:08.660700 [ 32432 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:08.664792 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:08.665455 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:08.672328 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:08.673047 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:08.685109 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:08.688041 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:08.688570 [ 32284 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:09.194651 [ 31868 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 6001ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:09.231439 [ 32081 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 32517ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:09.257207 [ 32087 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:09.257657 [ 32087 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:09.257790 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:09.257907 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:09.258151 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:09.258155 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:09.260421 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:09.260506 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:09.262122 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:09.262236 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:09.262357 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:09.262568 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:09.262674 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:09.262787 [ 31861 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:09.263211 [ 31861 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:09.263207 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:09.263389 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:09.263484 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:09.263522 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:09.263650 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:09.263672 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:09.263803 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:09.264260 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:09.265591 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:09.265672 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:09.265700 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:09.266340 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:09.266544 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:09.266889 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:09.267424 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:09.268485 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:09.268569 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:09.268919 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:09.269063 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:09.269021 [ 31857 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:09.269179 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:09.269487 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:09.269528 [ 31857 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:09.269787 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:09.269804 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:09.269997 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:09.271530 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:09.271594 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:09.273242 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:09.273291 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:09.273556 [ 31856 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:09.273892 [ 31856 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:09.273902 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:09.274177 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:09.274204 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:09.274392 [ 31877 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:09.274534 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:09.274549 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:09.274678 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:09.275957 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:09.276235 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:09.276838 [ 31877 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1207ms, ratio: 0.025641028, points per minute: 12.746161)
2026.05.06 08:49:09.277012 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:09.277102 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:09.277596 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:09.277606 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:09.277685 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:09.277779 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:09.278030 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:09.278249 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:09.278856 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:09.278984 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:09.279103 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:09.279572 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:09.280264 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:09.280442 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:09.280782 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:09.281260 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:09.281910 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:09.282452 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:09.282606 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:09.282713 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:09.284222 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:09.284374 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:09.289360 [ 31877 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:09.289458 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:09.289584 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:09.290809 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:09.292152 [ 31877 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:49:09.293557 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:09.293641 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:09.293768 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:09.293886 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:09.294207 [ 31877 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:09.299098 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:09.299291 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:09.299781 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:09.300372 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:09.301306 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:09.301763 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:09.306363 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:09.306534 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:09.306688 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:09.306972 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.307193 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.307356 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:09.307455 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:09.307712 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:09.308670 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:09.308801 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:09.308943 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:09.309202 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.309404 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.309574 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:09.309668 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:09.309880 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:09.310967 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:09.311096 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:09.311237 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:09.311488 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.311750 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.311912 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:09.312006 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:09.312198 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:09.313183 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:09.313308 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:09.313447 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:09.313717 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.313914 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.314082 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:09.314178 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:09.314367 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:09.315441 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:09.315588 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:09.315767 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:09.316026 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.316233 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.316389 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:09.316486 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:09.316700 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:09.317684 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:09.317808 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:09.317953 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:09.318206 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.318417 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.318586 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:09.318677 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:09.318871 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:09.319673 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:09.319799 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:09.319936 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:09.320187 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.320385 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:09.320545 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:09.320635 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:09.320829 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:09.321367 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:09.322073 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:09.322510 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:09.322602 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:09.326621 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:09.327975 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:09.328635 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:09.329119 [ 31845 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:09.329565 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:09.329609 [ 31845 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:09.329926 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:09.331788 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:09.335170 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:09.336594 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:09.336678 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:09.336751 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:09.338370 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:09.339567 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:09.339710 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:09.341553 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:09.343398 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:09.343481 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:09.343535 [ 31841 ] {} <Trace> test_15.t_r_ok (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 32676ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:09.343593 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:09.343790 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:09.347376 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:09.347524 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:09.348046 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:09.348711 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:09.349802 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:09.350120 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:09.353158 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:09.353276 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:09.353390 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:09.353602 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.353746 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.353857 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:09.353924 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:09.354059 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:09.354746 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:09.354835 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:09.354934 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:09.355103 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.355238 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.355343 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:09.355410 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:09.355553 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:09.356334 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:09.356424 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:09.356532 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:09.356696 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.356820 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.356923 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:09.356986 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:09.357107 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:09.357785 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:09.357873 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:09.357969 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:09.358131 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.358256 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.358361 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:09.358420 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:09.358548 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:09.359221 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:09.359303 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:09.359398 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:09.359570 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.359715 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.359816 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:09.359877 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:09.359993 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:09.360662 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:09.360752 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:09.360852 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:09.361024 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.361173 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.361278 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:09.361337 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:09.361460 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:09.361985 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:09.362069 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:09.362164 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:09.362321 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.362440 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:09.362545 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:09.362601 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:09.362716 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:09.363082 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:09.363553 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:09.363866 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:09.363937 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:09.366706 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:09.367448 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:09.367847 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:09.368208 [ 31839 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:09.368455 [ 31839 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:09.466668 [ 31837 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:09.472582 [ 31837 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:09.482137 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:09.483244 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:09.506605 [ 32418 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:09.506807 [ 32418 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:09.507224 [ 32418 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007054258 sec. (141.758 rows/sec., 3.32 KiB/sec.)
2026.05.06 08:49:09.507350 [ 32418 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:09.511774 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:09.512430 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:09.519264 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:09.521116 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:09.522393 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./src/Common/threadPoolCallbackRunner.h:0: std::future<std::shared_ptr<DB::MarksInCompressedFile>> std::__function::__policy_invoker<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>::__call_impl<std::__function::__default_alloc_func<std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority), std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)>>(std::__function::__policy_storage const*, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority&&) @ 0x0000000039d4c7c3
7. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x0000000039d41802
8. ./build_docker/./src/Storages/MergeTree/MergeTreeMarksLoader.cpp:73: DB::MergeTreeMarksLoader::startAsyncLoad() @ 0x0000000039d40c79
9. ./build_docker/./src/Storages/MergeTree/MergeTreeReaderCompact.cpp:0: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x0000000039dc48e8
10. ./src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:0: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x0000000039ad6f57
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x0000000039acf290
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:296: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool) @ 0x0000000039e50ea4
13. ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15007]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::shared_ptr<DB::AlterConversions const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&) @ 0x0000000039e5b639
14. ./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298: DB::createMergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::AlterConversions const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, std::shared_ptr<std::atomic<unsigned long>>, bool, bool, bool) @ 0x0000000039e578a6
15. ./contrib/llvm-project/libcxx/include/optional:260: DB::ReadFromPart::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000039e5f6a2
16. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:0: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000003bf82e23
17. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:296: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000003bff1445
18. ./build_docker/./src/Interpreters/MutationsInterpreter.cpp:1312: DB::MutationsInterpreter::addStreamsForLaterStages(std::vector<DB::MutationsInterpreter::Stage, std::allocator<DB::MutationsInterpreter::Stage>> const&, DB::QueryPlan&) const @ 0x00000000350d55d1
19. ./build_docker/./src/Interpreters/MutationsInterpreter.cpp:0: DB::MutationsInterpreter::execute() @ 0x00000000350dff6b
20. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::prepare() @ 0x0000000039fc79b2
21. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:2054: DB::MutateTask::execute() @ 0x0000000039fc1b76
22. ./build_docker/./src/Storages/MergeTree/MutatePlainMergeTreeTask.cpp:0: DB::MutatePlainMergeTreeTask::executeStep() @ 0x000000003a5099dc
23. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
24. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
25. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
26. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
27. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
28. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
29. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
30. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
31. ? @ 0x00007f6e2fbe4ac3
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:09.760026 [ 31818 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 33061ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:09.980426 [ 32317 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:09.980955 [ 32317 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: ::1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:09 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE730DE5C544E
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:09.981374 [ 32317 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:09.999412 [ 31808 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:10.005919 [ 31808 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:10.012705 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:10.013525 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:10.030132 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=784, jobs=664). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:10.059203 [ 31814 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 33359ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:10.251541 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=784, jobs=664). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202eb394
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202d9869
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202db514
6. ./src/Common/ThreadPool.h:0: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000003b01a5ac
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000003b01bc2b
8. ./build_docker/./src/Interpreters/MutationsInterpreter.cpp:213: DB::isStorageTouchedByMutations(std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::MergeTreeData::IMutationsSnapshot const>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::vector<DB::MutationCommand, std::allocator<DB::MutationCommand>> const&, std::shared_ptr<DB::Context const>) @ 0x00000000350a3d7a
9. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:2185: DB::MutateTask::prepare() @ 0x0000000039fc4116
10. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:2054: DB::MutateTask::execute() @ 0x0000000039fc1b76
11. ./build_docker/./src/Storages/MergeTree/MutatePlainMergeTreeTask.cpp:0: DB::MutatePlainMergeTreeTask::executeStep() @ 0x000000003a5099dc
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
13. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
15. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
16. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
17. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
18. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
19. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
20. ? @ 0x00007f6e2fbe4ac3
21. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:10.391182 [ 31806 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.391665 [ 31806 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.391685 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:10.391905 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:10.391956 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.392155 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.394562 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.394571 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.396455 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.396532 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.396775 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:10.396956 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:10.397064 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:10.397167 [ 31798 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.397600 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:10.397622 [ 31798 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.397880 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.397938 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:10.398003 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:10.398061 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:10.398182 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:10.398384 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.398837 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:10.399800 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.400329 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:10.400349 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.400990 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:10.401273 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.401227 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:10.401769 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:10.401966 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.402273 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:10.402344 [ 31796 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.402741 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:10.402792 [ 31796 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.403019 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:10.403120 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.403272 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.403866 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:10.404013 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:10.404127 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:10.405080 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.405227 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.406715 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.407080 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.407082 [ 31797 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.407570 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:10.407680 [ 31797 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.407977 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:10.407992 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.408294 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.409617 [ 31803 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:10.409723 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:10.409863 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.409874 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:10.410054 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.411672 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:10.411687 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.412378 [ 31803 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1118ms, ratio: 0.025641028, points per minute: 13.760838)
2026.05.06 08:49:10.412571 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:10.412602 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:10.412693 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:10.412703 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:10.412805 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:10.413391 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:10.413487 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:10.413600 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:10.414126 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:10.415632 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:10.415672 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:10.416406 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:10.416642 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:10.417073 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:10.417532 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:10.417908 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:10.418054 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:10.418877 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:10.419042 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:10.419158 [ 31803 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:10.422180 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:10.424639 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:10.424719 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:10.424838 [ 31803 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:10.424847 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:10.424951 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:10.424929 [ 31786 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 2003ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:10.425075 [ 31803 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:10.425008 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:10.427366 [ 31803 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:10.429875 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:10.430064 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:10.430480 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:10.431121 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:10.432160 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:10.432566 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:10.437292 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:10.437441 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:10.437610 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:10.437889 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.438117 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.438279 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:10.438375 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:10.438594 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:10.439522 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:10.439684 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:10.439830 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:10.440104 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.440313 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.440467 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:10.440574 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:10.440767 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:10.442011 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:10.442147 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:10.442304 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:10.442564 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.442763 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.442919 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:10.443014 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:10.443204 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:10.444229 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:10.444357 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:10.444512 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:10.444791 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.444990 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.445143 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:10.445234 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:10.445418 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:10.446626 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:10.446759 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:10.446904 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:10.447185 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.447394 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.447569 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:10.447693 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:10.447895 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:10.448912 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:10.449055 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:10.449197 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:10.449445 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.449659 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.449810 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:10.449900 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:10.450090 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:10.450879 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:10.450997 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:10.451131 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:10.451375 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.451588 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:10.451769 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:10.451859 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:10.452049 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:10.452578 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:10.453228 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:10.453658 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:10.453751 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:10.457901 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:10.459207 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:10.459883 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:10.460303 [ 31790 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:10.460608 [ 31790 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.460990 [ 31790 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:10.461047 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:10.461433 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.463450 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.465686 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:10.466609 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:10.466744 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:10.466854 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:10.469664 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:10.471982 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:10.472131 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:10.476121 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:10.478527 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:10.478624 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:10.478757 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:10.478883 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:10.483702 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:10.483904 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:10.484421 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:10.485034 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:10.485979 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:10.486386 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:10.491423 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:10.491659 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:10.491827 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:10.492128 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.492372 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.492558 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:10.492662 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:10.492881 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:10.493888 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:10.494027 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:10.494179 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:10.494459 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.494701 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.494890 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:10.494992 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:10.495205 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:10.496511 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:10.496661 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:10.496830 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:10.497106 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.497317 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.497484 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:10.497604 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:10.497812 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:10.498857 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:10.498990 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:10.499136 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:10.499400 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.499674 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.499837 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:10.499938 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:10.500141 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:10.501318 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:10.501466 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:10.501664 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:10.501957 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.502178 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.502343 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:10.502445 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:10.502686 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:10.503876 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:10.504020 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:10.504173 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:10.504443 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.504674 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.504838 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:10.504935 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:10.505141 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:10.505978 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:10.506107 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:10.506253 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:10.506533 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.506743 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:10.506900 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:10.506998 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:10.507204 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:10.507916 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:10.508645 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:10.509075 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:10.509171 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:10.512492 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:10.513874 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:10.514546 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:10.515013 [ 31784 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:10.515235 [ 31784 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:10.619981 [ 31776 ] {} <Trace> test_15.t (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 33953ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:10.637842 [ 31777 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.638258 [ 31777 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:10.638323 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:10.638705 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.641269 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.643202 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.643449 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:10.643663 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:10.643780 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:10.643782 [ 31773 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.644207 [ 31773 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.644230 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:10.644491 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:10.644611 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.644636 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:10.644759 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:10.644796 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.644889 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:10.645552 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:10.646752 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.646820 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.647071 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:10.647887 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:10.648095 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:10.648463 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.648507 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:10.648583 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.649059 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:10.649205 [ 31766 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.649637 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:10.649683 [ 31766 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.649982 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.650019 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:10.650437 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.650727 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:10.650919 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:10.651063 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:10.651959 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.652492 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.653891 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.654302 [ 31775 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.654678 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:10.654680 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.654977 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.655111 [ 31764 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.655421 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:10.655813 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.657285 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.657787 [ 31771 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:10.657901 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:10.657987 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.658073 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:10.659230 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.659752 [ 31760 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.660070 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:10.660167 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:10.660426 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.661130 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:10.661200 [ 31771 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 234ms, ratio: 0.025641028, points per minute: 65.74622)
2026.05.06 08:49:10.661246 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:10.661352 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:10.661428 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:10.661588 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:10.662237 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:10.662296 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.662352 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:10.662479 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:10.663035 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:10.664179 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.664259 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:10.664585 [ 31758 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.664782 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:10.664897 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:10.665042 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.665617 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:10.665830 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:10.666131 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:10.666535 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:10.666608 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:10.666678 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:10.667461 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.667999 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:10.668140 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:10.668255 [ 31771 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:10.669000 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:10.669328 [ 31757 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:10.669753 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:10.670089 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:10.670688 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:10.672200 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:10.673264 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:10.673362 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:10.673551 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:10.673708 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:10.673995 [ 31771 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:10.674099 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:10.674132 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:10.674220 [ 31771 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:10.674973 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:10.675085 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:10.675187 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:10.676592 [ 31771 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:10.678096 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:10.678579 [ 31771 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:10.678655 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:10.678857 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:10.679304 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:10.679903 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:10.680402 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:10.680564 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:10.680864 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:10.681275 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:10.684623 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:10.686637 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:10.686796 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:10.686953 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:10.687021 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:10.687069 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:10.687152 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:10.687231 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:10.687249 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.687470 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.687677 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:10.687779 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:10.687991 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:10.688990 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:10.689123 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:10.689271 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:10.689557 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.689769 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.689957 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:10.690086 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:10.690349 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:10.690884 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:10.691019 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:10.691527 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:10.691802 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:10.691948 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:10.692046 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:10.692099 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:10.692374 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.692605 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.692788 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:10.692889 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:10.692931 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:10.693097 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:10.693220 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:10.694137 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:10.694269 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:10.694416 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:10.694698 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.694910 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.695073 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:10.695172 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:10.695375 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:10.696103 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:10.696203 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:10.696308 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:10.696481 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.696540 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:10.696636 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.696676 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:10.696753 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:10.696822 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:10.696830 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:10.696952 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:10.697176 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.697452 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.697611 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:10.697711 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:10.697695 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:10.697829 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:10.697860 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:10.698029 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.698131 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:10.698176 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.698294 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:10.698366 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:10.698507 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:10.699257 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:10.699290 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:10.699387 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:10.699396 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:10.699487 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:10.699565 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:10.699699 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.699833 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.699859 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.699948 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:10.700011 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:10.700065 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.700132 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:10.700221 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:10.700314 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:10.700525 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:10.700785 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:10.700876 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:10.700969 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:10.701130 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.701265 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.701310 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:10.701384 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:10.701433 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:10.701450 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:10.701584 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:10.701591 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:10.701879 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.702082 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:10.702234 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:10.702320 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:10.702326 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:10.702413 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:10.702520 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:10.702537 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:10.702696 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.702840 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.702942 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:10.703005 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:10.703123 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:10.703127 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:10.703871 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:10.703911 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:10.703968 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:10.704070 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:10.704252 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.704388 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.704412 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:10.704508 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:10.704571 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:10.704564 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:10.704707 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:10.705236 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:10.705328 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:10.705426 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:10.705607 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.705744 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:10.705852 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:10.705917 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:10.706042 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:10.706418 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:10.706966 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:10.707239 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:10.707293 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:10.709474 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:10.710471 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:10.711028 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:10.711129 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:10.711651 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:10.711771 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:10.712056 [ 31753 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:10.712243 [ 31753 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:11.146345 [ 31738 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:11.153549 [ 31738 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:11.163518 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:11.164675 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:11.178850 [ 31731 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:11.179248 [ 31731 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:11.179290 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:11.179693 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:11.179734 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:11.180111 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:11.182483 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:11.182574 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:11.184428 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:11.184562 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:11.184777 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:11.184961 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:11.185061 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:11.185159 [ 31729 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:11.185492 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:11.185585 [ 31729 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:11.185845 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:11.185889 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:11.185961 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:11.186006 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:11.186124 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:11.186316 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:11.186720 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:11.188205 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:11.188320 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:11.188543 [ 32387 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:11.188675 [ 32387 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:11.188776 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:11.188984 [ 32387 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006931279 sec. (144.274 rows/sec., 3.38 KiB/sec.)
2026.05.06 08:49:11.189025 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:11.189088 [ 32387 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:11.189243 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:11.189535 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:11.189842 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:11.189994 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:11.191255 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:11.191371 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:11.191426 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:11.191587 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:11.191760 [ 31725 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:11.192163 [ 31725 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:11.192161 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:11.192414 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:11.192488 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:11.192711 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:11.193730 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:11.194220 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:11.194290 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:11.194527 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:11.195758 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:11.195823 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:11.196223 [ 31720 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:11.196554 [ 31720 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:11.196608 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:11.196822 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:11.197016 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:11.197136 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:11.198288 [ 31730 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:11.198410 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:11.198620 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:11.198821 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:11.198888 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:11.200363 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:11.200493 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:11.201211 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:11.201323 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:11.201397 [ 31730 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 523ms, ratio: 0.025641028, points per minute: 29.416094)
2026.05.06 08:49:11.201427 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:11.201594 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:11.201687 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:11.202274 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:11.202362 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:11.202454 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:11.202607 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:11.202945 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:11.203281 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:11.204277 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:11.204325 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:11.204992 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:11.205163 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:11.205456 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:11.205948 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:11.206702 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:11.206852 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:11.207246 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:11.207367 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:11.207469 [ 31730 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:11.210926 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:11.212656 [ 31730 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:11.212755 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:11.212887 [ 31730 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:11.213647 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:11.213738 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:11.213869 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:11.213991 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:11.215580 [ 31730 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:11.217085 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:11.218808 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:11.219008 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:11.219565 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:11.220184 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:11.220787 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:11.221216 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:11.221354 [ 32282 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:11.221641 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:11.226363 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:11.226534 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:11.226690 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:11.226978 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.227198 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.227362 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:11.227461 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:11.227733 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:11.228705 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:11.228834 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:11.228977 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:11.229237 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.229456 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.229627 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:11.229721 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:11.229918 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:11.231019 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:11.231148 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:11.231290 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:11.231554 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.231785 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.231946 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:11.232042 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:11.232234 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:11.233284 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:11.233414 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:11.233575 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:11.233849 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.234051 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.234212 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:11.234309 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:11.234520 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:11.235684 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:11.235820 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:11.235972 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:11.236236 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.236446 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.236625 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:11.236724 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:11.236929 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:11.237938 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:11.238079 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:11.238224 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:11.238479 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.238697 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.238852 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:11.238944 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:11.239138 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:11.239971 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:11.240099 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:11.240239 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:11.240508 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.240707 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:11.240856 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:11.240948 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:11.241141 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:11.241726 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:11.242421 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:11.242856 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:11.242951 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:11.246798 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:11.248306 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:11.249022 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:11.249419 [ 31714 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:11.249704 [ 31714 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:11.250055 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:11.250067 [ 31714 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:11.250390 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:11.252559 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:11.254551 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:11.255424 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:11.255565 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:11.255700 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:11.258482 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:11.261057 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:11.261245 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:11.265360 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:11.267715 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:11.267812 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:11.267964 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:11.268092 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:11.272909 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:11.273115 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:11.273593 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:11.274000 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:11.275054 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:11.275440 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:11.280076 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:11.280228 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:11.280381 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:11.280702 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.280967 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.281133 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:11.281232 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:11.281442 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:11.282381 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:11.282527 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:11.282678 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:11.282971 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.283195 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.283355 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:11.283450 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:11.283707 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:11.284792 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:11.284923 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:11.285069 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:11.285330 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.285546 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.285707 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:11.285803 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:11.285999 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:11.287026 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:11.287152 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:11.287294 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:11.287583 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.287820 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.287978 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:11.288073 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:11.288265 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:11.289336 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:11.289463 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:11.289627 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:11.289888 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.290099 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.290258 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:11.290353 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:11.290569 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:11.291529 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:11.291715 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:11.291863 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:11.292125 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.292331 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.292487 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:11.292595 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:11.292793 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:11.293586 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:11.293711 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:11.293852 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:11.294108 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.294319 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:11.294470 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:11.294578 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:11.294773 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:11.295305 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:11.296016 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:11.296418 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:11.296525 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:11.300352 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:11.301711 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:11.302362 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:11.302664 [ 31705 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:11.545372 [ 31695 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:11.549346 [ 31695 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:11.558867 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:11.559996 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:11.583068 [ 32376 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:11.583218 [ 32376 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:11.583554 [ 32376 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007094759 sec. (140.949 rows/sec., 3.30 KiB/sec.)
2026.05.06 08:49:11.583681 [ 32376 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:11.588063 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:11.588785 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:11.595813 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:11.596578 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:11.607418 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:11.610694 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:11.611034 [ 32280 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:12.093651 [ 31900 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1668ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:12.205881 [ 32071 ] {} <Trace> test_1.test_without_merge (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 35525ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:12.264259 [ 32154 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.264698 [ 32154 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.264748 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:12.265026 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.265065 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.265377 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.267869 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.267971 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.269737 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.269811 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.269949 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:12.270048 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:12.270119 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:12.270218 [ 32130 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.270676 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:12.270720 [ 32130 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.270788 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:12.270889 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:12.270981 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:12.271059 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.271104 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.271395 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:12.271416 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.272794 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:12.272894 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.273223 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.273427 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:12.273596 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:12.273914 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:12.274390 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.275446 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.275636 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:12.275754 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:12.275833 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:12.275825 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.276207 [ 32143 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.276568 [ 32143 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.276568 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:12.276822 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.276910 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.277059 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.278621 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.278663 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.278729 [ 32160 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:12.278794 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:12.278903 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:12.280293 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:12.280335 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.281018 [ 32160 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1066ms, ratio: 0.025641028, points per minute: 14.432098)
2026.05.06 08:49:12.281147 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:12.281214 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:12.281219 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:12.281327 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:12.281428 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:12.281847 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:12.281921 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:12.281989 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:12.282315 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:12.283118 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:12.283527 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:12.284116 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:12.284228 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:12.284415 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:12.284524 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:12.284533 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:12.284977 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.286151 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:12.286246 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:12.286313 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:12.286876 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:12.289187 [ 32160 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:12.289253 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:12.289336 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:12.289734 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:12.289814 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:12.289956 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:12.290075 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:12.291748 [ 32160 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 10ms, ratio: 0.025641028, points per minute: 1538.4617)
2026.05.06 08:49:12.294769 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:12.294954 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:12.295397 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:12.295983 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.296920 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:12.297313 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:12.302141 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:12.302290 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:12.302445 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:12.302750 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.302973 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.303139 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:12.303238 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:12.303451 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:12.304522 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:12.304676 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:12.304826 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:12.305095 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.305307 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.305468 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:12.305581 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:12.305783 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:12.306921 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:12.307052 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:12.307198 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.307480 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.307738 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.307906 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.308004 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.308204 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.309206 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:12.309333 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:12.309478 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:12.309757 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.309960 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.310120 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:12.310216 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:12.310412 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:12.311485 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:12.311689 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:12.311863 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:12.312160 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.312420 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.312621 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:12.312747 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:12.312982 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:12.314099 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:12.314233 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:12.314388 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.314705 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.314927 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.315092 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.315189 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.315394 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.316266 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:12.316395 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:12.316560 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:12.316827 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.317036 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:12.317207 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:12.317302 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:12.317516 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:12.318061 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:12.318853 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:12.319315 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:12.319412 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:12.323275 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:12.324665 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.325319 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.325762 [ 31684 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:12.326006 [ 31684 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.326315 [ 31684 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:12.326384 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.326713 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.328759 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.330452 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:12.331420 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:12.331520 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:12.331596 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:12.334837 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:12.337694 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:12.337883 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:12.343038 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:12.343102 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:12.343162 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:12.343252 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:12.343334 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:12.346979 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:12.347108 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:12.347697 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:12.348317 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.349536 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:12.349828 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:12.352417 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:12.352540 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:12.352654 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:12.352856 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.353006 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.353127 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:12.353200 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:12.353335 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:12.353969 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:12.354073 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:12.354173 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:12.354346 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.354479 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.354595 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:12.354659 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:12.354778 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:12.355514 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:12.355628 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:12.355728 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.355889 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.356017 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.356119 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.356182 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.356301 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.356950 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:12.357032 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:12.357120 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:12.357270 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.357384 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.357478 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:12.357549 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:12.357659 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:12.358318 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:12.358401 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:12.358508 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:12.358670 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.358803 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.358904 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:12.358976 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:12.359098 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:12.359760 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:12.359847 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:12.359944 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.360106 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.360231 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.360330 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.360389 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.360519 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.361009 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:12.361087 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:12.361177 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:12.361334 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.361455 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:12.361564 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:12.361621 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:12.361738 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:12.362102 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:12.362578 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:12.362842 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:12.362904 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:12.366130 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:12.366924 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.367238 [ 32070 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:12.367317 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.367684 [ 32072 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:12.367891 [ 32072 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:12.373641 [ 32070 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:12.382986 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:12.384149 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:12.411065 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.411727 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.562459 [ 31904 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.562892 [ 31904 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.562974 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:12.563280 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.563386 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.563694 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.566235 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.566316 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.568035 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.568110 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.568353 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:12.568604 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:12.568744 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:12.568828 [ 31887 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.569270 [ 31887 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.569288 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:12.569552 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.569599 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:12.569661 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.569711 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:12.569818 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.569827 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:12.570518 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:12.571544 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.571595 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.572019 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:12.572873 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:12.573157 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:12.573551 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:12.573584 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.573782 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.574026 [ 32127 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.574079 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.574417 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:12.574469 [ 32127 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.574758 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.574759 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.575045 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.575645 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:12.575811 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:12.575925 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:12.576835 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.576858 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.578600 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.578791 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.579193 [ 32133 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.579574 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.579688 [ 32133 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.579952 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.580017 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:12.580311 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.581536 [ 32185 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:12.581637 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:12.581676 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.581789 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:12.582181 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.583398 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.583756 [ 32102 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.584001 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.584079 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:12.584164 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.584292 [ 32185 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 293ms, ratio: 0.025641028, points per minute: 52.507225)
2026.05.06 08:49:12.584475 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:12.584587 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:12.585149 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:12.585208 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:12.585247 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:12.585334 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:12.585349 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:12.585475 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:12.585775 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.585893 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:12.587209 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:12.587403 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.587767 [ 31895 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.587981 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:12.588104 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.588172 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:12.588271 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:12.588372 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.588572 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:12.589097 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.590479 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:12.590535 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:12.590648 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:12.590681 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:12.590763 [ 32185 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:12.591055 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.592805 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.593192 [ 32064 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.593566 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.593897 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.594675 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:12.595897 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.596319 [ 32185 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:12.596418 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:12.596560 [ 32185 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:12.597331 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:12.597432 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:12.597593 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:12.597743 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:12.597948 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:12.598740 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:12.598861 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:12.598972 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:12.599133 [ 32185 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:12.600706 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:12.601525 [ 32185 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:12.601972 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:12.602081 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:12.603082 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:12.603272 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:12.603747 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:12.604277 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.604332 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:12.605300 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:12.605730 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:12.606940 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:12.607016 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:12.607156 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:12.607272 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:12.610677 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:12.610828 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:12.610981 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:12.611265 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.611482 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.611700 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:12.611800 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:12.612009 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:12.612043 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:12.612252 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:12.612750 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:12.613021 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:12.613154 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:12.613256 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.613299 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:12.613623 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.613855 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.614055 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:12.614171 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:12.614260 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:12.614371 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:12.614669 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:12.615473 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:12.615658 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:12.615807 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.616069 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.616273 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.616433 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.616546 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.616737 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.617751 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:12.617879 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:12.618021 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:12.618273 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.618471 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.618644 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:12.618738 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:12.618927 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:12.619298 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:12.619447 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:12.619652 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:12.619944 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.620069 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:12.620201 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.620217 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:12.620392 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:12.620410 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:12.620536 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:12.620751 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:12.620765 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.620993 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.621156 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:12.621254 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:12.621453 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:12.621719 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:12.621850 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:12.621998 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:12.622274 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.622456 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:12.622479 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.622604 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:12.622669 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:12.622754 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.622766 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:12.622962 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:12.623015 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.623218 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.623371 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.623460 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.623707 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.624109 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:12.624242 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:12.624387 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.624466 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:12.624607 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:12.624661 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.624743 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:12.624875 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.624999 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.625046 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.625143 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.625198 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:12.625337 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.625345 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:12.625474 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:12.625686 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:12.626212 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:12.626411 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:12.626583 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:12.626773 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:12.626941 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:12.627073 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.627277 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.627348 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:12.627435 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:12.627441 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:12.627563 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:12.627784 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:12.628891 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:12.629019 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:12.629166 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:12.629422 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.629645 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.629802 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:12.629898 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:12.630108 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:12.631107 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:12.631265 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:12.631289 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:12.631422 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.631720 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.631925 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.632080 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.632170 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.632365 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.632668 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.633145 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:12.633269 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:12.633311 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.633408 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:12.633685 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.633812 [ 32156 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:12.633901 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:12.634120 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:12.634213 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:12.634408 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:12.635038 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:12.635795 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:12.636212 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:12.636305 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:12.640325 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:12.641847 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.642638 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.643086 [ 32155 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:12.643278 [ 32155 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:12.663219 [ 32161 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:12.667191 [ 32161 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:12.672954 [ 32137 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 5320ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:12.676290 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:12.677336 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:12.700549 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:12.700680 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:12.700963 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007272879 sec. (137.497 rows/sec., 3.22 KiB/sec.)
2026.05.06 08:49:12.701047 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:12.704749 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:12.705347 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:12.712001 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:12.712683 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:12.725005 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.727973 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.728448 [ 32285 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:12.825959 [ 31863 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 36099ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:12.831794 [ 32083 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.832163 [ 32083 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.832197 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:12.832463 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.832508 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.832759 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.834656 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.834769 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.836577 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.836648 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.836805 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:12.837016 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:12.837118 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:12.837185 [ 32186 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.837562 [ 32186 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.837579 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:12.837787 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.837865 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:12.837907 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.837989 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:12.838056 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.838113 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:12.838744 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:12.839816 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.839911 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.840141 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:12.840699 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:12.840920 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:12.841053 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.841096 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.841249 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:12.841605 [ 32159 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.841780 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.841925 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:12.842035 [ 32159 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.842216 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.842347 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.842669 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.842936 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:12.843071 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:12.843183 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:12.843829 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.844265 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.845234 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.845420 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.845638 [ 32145 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.846022 [ 32145 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.846033 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:12.846219 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.846342 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.846332 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.847912 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.848131 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.848366 [ 32164 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:12.848465 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:12.848627 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:12.849872 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.850084 [ 32098 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.850144 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:12.850328 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.850622 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.850933 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:12.851044 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:12.851146 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:12.851213 [ 32164 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 250ms, ratio: 0.025641028, points per minute: 61.538467)
2026.05.06 08:49:12.851446 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:12.851563 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:12.852169 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:12.852274 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.852275 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:12.852407 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:12.853001 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:12.853997 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.854388 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:12.854400 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:12.854399 [ 32188 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.854827 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.855089 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:12.855099 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.855287 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:12.855650 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:12.856122 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.856800 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:12.856950 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:12.857248 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:12.857350 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.857381 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:12.857492 [ 32164 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:12.859151 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:12.859584 [ 32141 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:12.859943 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:12.860235 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:12.861106 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:12.862185 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:12.862819 [ 32164 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:12.862925 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:12.863051 [ 32164 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:12.863411 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:12.863489 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:12.863667 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:12.863787 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:12.864175 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:12.864989 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:12.865055 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:12.865112 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:12.865706 [ 32164 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:12.866654 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:12.867949 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:12.867988 [ 32164 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:12.868045 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:12.868238 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:12.868422 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:12.868853 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:12.869525 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.870208 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:12.870347 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:12.870846 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:12.871229 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:12.871270 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:12.871337 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:12.871399 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:12.873863 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:12.874051 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:12.874566 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:12.875101 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:12.875466 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:12.875670 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:12.875827 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:12.875913 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:12.876115 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.876306 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:12.876331 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.876493 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:12.876614 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:12.876836 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:12.877781 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:12.877906 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:12.878049 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:12.878304 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.878525 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.878690 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:12.878807 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:12.879002 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:12.880118 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:12.880249 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:12.880390 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.880654 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.880850 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.880915 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:12.881025 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.881062 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:12.881121 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.881214 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:12.881310 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.881493 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.881728 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.881888 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:12.881984 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:12.882200 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:12.882295 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:12.882420 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:12.882575 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:12.882854 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.883042 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:12.883067 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.883137 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:12.883225 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:12.883223 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:12.883318 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:12.883381 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.883521 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.883526 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:12.883631 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:12.883689 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:12.883794 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:12.884451 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:12.884547 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:12.884633 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.884753 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:12.884780 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.884906 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:12.884918 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.885035 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.885072 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:12.885087 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.885206 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.885353 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.885616 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.885745 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:12.885814 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:12.885801 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:12.885891 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:12.885919 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:12.886021 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.886134 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.886162 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:12.886221 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:12.886279 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:12.886383 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:12.887017 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:12.887094 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:12.887176 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:12.887322 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.887359 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:12.887438 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.887541 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:12.887539 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:12.887596 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:12.887730 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:12.887777 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.888078 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.888306 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.888351 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:12.888430 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:12.888489 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.888528 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:12.888630 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.888716 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.888834 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.888862 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.888927 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:12.888991 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:12.889102 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:12.889543 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:12.889610 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:12.889686 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:12.889721 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:12.889818 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.889849 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:12.889919 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:12.889996 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:12.889992 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:12.890043 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:12.890139 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:12.890251 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.890435 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:12.890454 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:12.890624 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:12.890750 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:12.890851 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:12.890961 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:12.891064 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:12.891112 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:12.891537 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:12.892299 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:12.892732 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:12.892826 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:12.894321 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:12.894987 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.895300 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.895547 [ 32182 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:12.895780 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:12.897100 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.897769 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:12.898161 [ 32172 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:13.014878 [ 32151 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:13.021300 [ 32151 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:13.028258 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:13.029095 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:13.044183 [ 32451 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:13.044264 [ 32451 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:13.044420 [ 32451 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.004021225 sec. (248.680 rows/sec., 5.83 KiB/sec.)
2026.05.06 08:49:13.044466 [ 32451 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:13.047185 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:13.047590 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:13.051447 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:13.051945 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:13.059231 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:13.061359 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:13.061772 [ 32284 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:13.339790 [ 32180 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9568ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:13.401628 [ 32135 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1308ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:13.602618 [ 32052 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:13.603016 [ 32043 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 36923ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:13.607023 [ 32052 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:13.616332 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:13.617349 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:13.640451 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:13.641034 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:13.752353 [ 32044 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 7569ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:13.785776 [ 31937 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:13.786209 [ 31937 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:13.786209 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:13.786595 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:13.786636 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:13.787039 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:13.788704 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:13.789004 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:13.790486 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:13.790783 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:13.790955 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:13.791030 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:13.791069 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:13.791578 [ 32032 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:13.792105 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:13.792297 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:13.792589 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:13.793256 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:13.793328 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:13.793329 [ 32032 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:13.793575 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:13.793724 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:13.794073 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:13.795090 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:13.795299 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:13.795852 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:13.795867 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:13.796057 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:13.796487 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:13.797019 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:13.798195 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:13.798632 [ 32029 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:13.799088 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:13.799357 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:13.799394 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:13.799572 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:13.799598 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:13.799770 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:13.800197 [ 32019 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:13.800540 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:13.800746 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:13.804670 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:13.804760 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:13.805866 [ 32051 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:13.805974 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:13.806124 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:13.806137 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:13.806472 [ 32037 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:13.806566 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:13.806809 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:13.807089 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:13.807713 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:13.807782 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:13.807840 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:13.808914 [ 32051 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 941ms, ratio: 0.025641028, points per minute: 16.34922)
2026.05.06 08:49:13.808929 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:13.809124 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:13.809220 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:13.809865 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:13.809957 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:13.810051 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:13.810544 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:13.810662 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:13.810982 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:13.811387 [ 32047 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:13.811770 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:13.811986 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:13.812101 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:13.812920 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:13.813108 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:13.813163 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:13.813314 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:13.813738 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:13.813841 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:13.814239 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:13.815837 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:13.816048 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:13.816191 [ 32051 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:13.816342 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:13.818168 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:13.820649 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:13.820729 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:13.820855 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:13.820970 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:13.822298 [ 32051 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:13.822402 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:13.822547 [ 32051 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:13.825734 [ 32051 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:49:13.825781 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:13.825971 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:13.826421 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:13.826979 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:13.827974 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:13.828373 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:13.829259 [ 32051 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 4ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:13.833065 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:13.833216 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:13.833369 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:13.833676 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.833892 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.834054 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:13.834152 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:13.834360 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:13.835307 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:13.835439 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:13.835601 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:13.835906 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.836112 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.836288 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:13.836381 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:13.836589 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:13.837683 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:13.837809 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:13.837949 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:13.838200 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.838397 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.838567 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:13.838663 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:13.838854 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:13.839888 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:13.840017 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:13.840155 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:13.840407 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.840634 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.840792 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:13.840887 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:13.841081 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:13.842178 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:13.842306 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:13.842449 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:13.842725 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.842932 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.843087 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:13.843183 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:13.843383 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:13.844403 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:13.844564 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:13.844709 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:13.844963 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.845163 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.845317 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:13.845407 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:13.845614 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:13.846366 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:13.846488 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:13.846642 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:13.846891 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.847089 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:13.847237 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:13.847327 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:13.847531 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:13.848112 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:13.848813 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:13.849225 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:13.849318 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:13.853222 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:13.854705 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:13.855395 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:13.855899 [ 32014 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:13.856271 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:13.856331 [ 32014 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:13.856574 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:13.858277 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:13.860136 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:13.860836 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:13.860894 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:13.860950 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:13.863796 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:13.866292 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:13.866451 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:13.870856 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:13.873485 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:13.873597 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:13.873728 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:13.873851 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:13.880033 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:13.880235 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:13.880858 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:13.881544 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:13.882754 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:13.883156 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:13.888397 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:13.888605 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:13.888765 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:13.889062 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.889281 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.889494 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:13.889622 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:13.889837 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:13.890881 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:13.891020 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:13.891173 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:13.891492 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.891779 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.891962 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:13.892062 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:13.892267 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:13.893539 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:13.893683 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:13.893832 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:13.894100 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.894304 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.894537 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:13.894644 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:13.894845 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:13.895959 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:13.896098 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:13.896243 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:13.896567 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.896800 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.896960 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:13.897057 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:13.897254 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:13.898450 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:13.898609 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:13.898761 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:13.899030 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.899240 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.899426 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:13.899566 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:13.899801 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:13.900876 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:13.901010 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:13.901157 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:13.901459 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.901699 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.901855 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:13.901947 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:13.902175 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:13.903042 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:13.903173 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:13.903317 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:13.903708 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.903931 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:13.904086 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:13.904179 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:13.904397 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:13.904985 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:13.905755 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:13.906174 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:13.906266 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:13.910538 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:13.912184 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:13.912981 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:13.913438 [ 32012 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:13.913666 [ 32012 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:14.128202 [ 32013 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:14.136997 [ 32013 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:14.146737 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:14.147842 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:14.170671 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:14.170825 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:14.171213 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007108209 sec. (140.682 rows/sec., 3.30 KiB/sec.)
2026.05.06 08:49:14.171336 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:14.176078 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:14.176784 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:14.184018 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:14.184756 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:14.197378 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:14.200867 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:14.201493 [ 32280 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:14.716098 [ 31996 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:14.722466 [ 31996 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:14.731748 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:14.732825 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:14.755278 [ 32424 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:14.755445 [ 32424 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:14.755853 [ 32424 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007072299 sec. (141.397 rows/sec., 3.31 KiB/sec.)
2026.05.06 08:49:14.755953 [ 32424 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:14.760233 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:14.760885 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:14.767675 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:14.768323 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:14.780682 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:14.783539 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:14.784026 [ 32271 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:14.911576 [ 31991 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 8588ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:14.988200 [ 31982 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:14.988678 [ 31982 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:14.988735 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:14.989023 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:14.989181 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:14.989336 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:14.991847 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:14.991945 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:14.994138 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:14.994178 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:14.994429 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:14.994663 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:14.994777 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:14.994783 [ 31978 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:14.995181 [ 31978 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:14.995203 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:14.995439 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:14.995538 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:14.995669 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:14.995668 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:14.995762 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:14.995805 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:14.996409 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:14.997479 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:14.997641 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:14.998038 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:14.998720 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:14.998951 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:14.999106 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:14.999207 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:14.999348 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:14.999707 [ 31977 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:14.999932 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:15.000021 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:15.000081 [ 31977 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:15.000333 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:15.000355 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:15.000617 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:15.001351 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:15.001539 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:15.001661 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:15.002069 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:15.002163 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:15.004006 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:15.004091 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:15.004405 [ 31971 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:15.004834 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:15.004897 [ 31971 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:15.005176 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:15.005211 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:15.005478 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:15.006871 [ 31985 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:15.006977 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:15.007117 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:15.007132 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:15.007525 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:15.008827 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:15.009253 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:15.009783 [ 31985 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1180ms, ratio: 0.025641028, points per minute: 13.03781)
2026.05.06 08:49:15.009967 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:15.010069 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:15.010237 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:15.010362 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:15.010485 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:15.010709 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:15.010806 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:15.010910 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:15.011420 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:15.012995 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:15.013303 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:15.013701 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:15.013884 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:15.014217 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:15.014636 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:15.015586 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:15.015764 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:15.015960 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:15.016090 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:15.016196 [ 31985 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:15.019787 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:15.021350 [ 31985 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:15.021444 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:15.021581 [ 31985 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:15.022445 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:15.022547 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:15.022669 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:15.022785 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:15.023998 [ 31985 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:15.027643 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:15.027830 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:15.028260 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:15.028774 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:15.029748 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:15.030121 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:15.034633 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:15.034796 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:15.034949 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:15.035232 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.035448 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.035664 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:15.035765 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:15.035973 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:15.036911 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:15.037043 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:15.037188 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:15.037445 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.037666 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.037821 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:15.037912 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:15.038105 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:15.039191 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:15.039319 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:15.039460 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:15.039769 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.039974 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.040134 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:15.040229 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:15.040420 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:15.041444 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:15.041591 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:15.041731 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:15.041982 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.042176 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.042328 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:15.042435 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:15.042643 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:15.043769 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:15.043904 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:15.044048 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:15.044305 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.044529 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.044686 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:15.044781 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:15.044977 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:15.045952 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:15.046076 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:15.046219 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:15.046473 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.046693 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.046861 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:15.046950 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:15.047144 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:15.047957 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:15.048083 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:15.048219 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:15.048467 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.048685 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:15.048832 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:15.048924 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:15.049115 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:15.049654 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:15.050311 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:15.050736 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:15.050827 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:15.054555 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:15.055896 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:15.056539 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:15.057068 [ 31963 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:15.057462 [ 31963 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:15.057543 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:15.057862 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:15.059888 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:15.061575 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:15.062364 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:15.062466 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:15.062581 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:15.065392 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:15.067764 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:15.067923 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:15.071957 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:15.074285 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:15.074359 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:15.074474 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:15.074603 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:15.077426 [ 32318 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:15.078016 [ 32318 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:15 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE7320E2A97E4
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:15.078464 [ 32318 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:15.079282 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:15.079459 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:15.080021 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:15.080593 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:15.081557 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:15.081920 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:15.086754 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:15.086917 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:15.087076 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:15.087365 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.087650 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.087832 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:15.087937 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:15.088150 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:15.089265 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:15.089404 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:15.089566 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:15.089828 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.090029 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.090208 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:15.090299 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:15.090491 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:15.091725 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:15.091883 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:15.092047 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:15.092333 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.092583 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.092768 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:15.092877 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:15.093097 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:15.094137 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:15.094266 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:15.094406 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:15.094683 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.094899 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.095055 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:15.095149 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:15.095339 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:15.096455 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:15.096482 [ 31640 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #5
2026.05.06 08:49:15.096600 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:49:15.096606 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:15.096813 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:15.096942 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:15.097073 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.097283 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.097442 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:15.097566 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:15.097767 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:15.098774 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:15.098917 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:15.099060 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:15.099315 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.099531 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.099720 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:15.099811 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:15.100004 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:15.100781 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:15.100904 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:15.101045 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:15.101297 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.101495 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:15.101664 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:15.101756 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:15.101946 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:15.102487 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:15.103168 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:15.103592 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:15.103785 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:15.107335 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:15.108684 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:15.109330 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:15.109791 [ 31964 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:15.110008 [ 31964 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:15.285480 [ 31959 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:15.291775 [ 31959 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:15.298199 [ 31953 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1897ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:15.301330 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:15.302434 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:15.329374 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:15.329940 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:15.556649 [ 31948 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:15.562722 [ 31948 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:15.571555 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:15.572611 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:15.594228 [ 32340 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:15.594358 [ 32340 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:15.594670 [ 32340 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006481598 sec. (154.283 rows/sec., 3.62 KiB/sec.)
2026.05.06 08:49:15.594753 [ 32340 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:15.598762 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:15.599356 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:15.606058 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:15.606797 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:15.618777 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:15.621622 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:15.622119 [ 32282 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:15.639804 [ 31939 ] {} <Trace> test_1.t (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 32121ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:16.031761 [ 31936 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.032272 [ 31936 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.032337 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:16.032646 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.032723 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.033064 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.035219 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.035369 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.037055 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.037141 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.037370 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:16.037642 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:16.037757 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:16.037816 [ 31929 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.038237 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:16.038250 [ 31929 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.038562 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:16.038596 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.038611 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.038699 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:16.038832 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:16.038907 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.039438 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:16.040560 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.040685 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.041027 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:16.041826 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:16.042137 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:16.042335 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.042549 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:16.042637 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.042785 [ 31924 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.043014 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:16.043198 [ 31924 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.043203 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:16.043551 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.043562 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.043950 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.044610 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:16.044767 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:16.044883 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:16.045593 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.045670 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.047360 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.047450 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.047812 [ 32131 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.048157 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:16.048223 [ 32131 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.048467 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.048532 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.048827 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.050247 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.050281 [ 31933 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:16.050404 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:16.050615 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.050636 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:16.051965 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:16.052292 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.052630 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:16.052645 [ 31921 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.052711 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:16.052782 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:16.053022 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.053036 [ 31933 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1029ms, ratio: 0.025641028, points per minute: 14.951036)
2026.05.06 08:49:16.053249 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:16.053295 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.053354 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:16.053989 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:16.054088 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:16.054191 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:16.054727 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:16.055049 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.055414 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:16.056058 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:16.056702 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.056821 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:16.057008 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:16.057062 [ 31914 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.057331 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:16.057450 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.057782 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:16.057813 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.057872 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:16.058046 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:16.058965 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:16.059094 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:16.059198 [ 31933 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:16.059635 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.061375 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.061819 [ 31915 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.062245 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.062475 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:16.062556 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:16.062618 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:16.062629 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.062718 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:16.062825 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:16.064396 [ 31933 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:16.064492 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:16.064623 [ 31933 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:16.064626 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.066472 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:16.066613 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:16.066772 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:16.066966 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:16.067256 [ 31933 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:16.067463 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:16.068561 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:16.068854 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:16.068889 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:16.069008 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:16.069109 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:16.069917 [ 31933 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:16.071326 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:16.071435 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:16.071554 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:16.071770 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.071925 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.072030 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:16.072096 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:16.072189 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:16.072231 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:16.072913 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:16.073008 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:16.073105 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:16.073283 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.073415 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.073530 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:16.073596 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:16.073725 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:16.074444 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:16.074554 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:16.074624 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:16.074649 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:16.074801 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:16.074856 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.074987 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.075092 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:16.075155 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:16.075276 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:16.075974 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:16.076064 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:16.076159 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:16.076325 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.076449 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.076561 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:16.076635 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:16.076756 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:16.077448 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:16.077547 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:16.077641 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:16.077803 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.077935 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.078034 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:16.078096 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:16.078220 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:16.078887 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:16.078976 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:16.079077 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:16.079159 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:16.079244 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.079391 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.079516 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:16.079577 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:16.079714 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:16.080243 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:16.080323 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:16.080422 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:16.080588 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.080718 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:16.080811 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:16.080868 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:16.080985 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:16.081363 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:16.081586 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:16.081680 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:16.081820 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:16.081854 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:16.081950 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:16.082126 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:16.082190 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:16.085280 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:16.085966 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.086323 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.086665 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:16.086723 [ 32122 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:16.086849 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:16.087243 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:16.087736 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:16.088733 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:16.089115 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:16.093872 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:16.094023 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:16.094173 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:16.094471 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.094705 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.094866 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:16.094962 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:16.095167 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:16.096134 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:16.096269 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:16.096413 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:16.096702 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.096903 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.097058 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:16.097171 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:16.097399 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:16.098495 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:16.098639 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:16.098781 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:16.099035 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.099233 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.099389 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:16.099483 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:16.099721 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:16.100716 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:16.100857 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:16.100997 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:16.101303 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.101538 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.101700 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:16.101795 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:16.101990 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:16.103074 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:16.103203 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:16.103347 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:16.103668 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.103880 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.104037 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:16.104134 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:16.104345 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:16.105356 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:16.105544 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:16.105740 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:16.106063 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.106310 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.106481 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:16.106594 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:16.106792 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:16.107697 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:16.107838 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:16.107984 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:16.108259 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.108548 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:16.108751 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:16.108868 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:16.109071 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:16.109664 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:16.110353 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:16.110777 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:16.110870 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:16.114982 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:16.116335 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.116996 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.117390 [ 31906 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:16.117587 [ 31906 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:16.246432 [ 32118 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:16.252785 [ 32118 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:16.262767 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:16.263942 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:16.286573 [ 32463 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:16.286709 [ 32463 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:16.287006 [ 32463 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006407778 sec. (156.060 rows/sec., 3.66 KiB/sec.)
2026.05.06 08:49:16.287132 [ 32463 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:16.292391 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:16.293146 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:16.300808 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:16.301624 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:16.314043 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.317443 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.318020 [ 32274 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:16.338054 [ 31923 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 8914ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:16.393515 [ 31903 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1095ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:16.528016 [ 31907 ] {} <Trace> test_15.src (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 39848ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:16.591159 [ 31897 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:16.591210 [ 32115 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.591755 [ 32115 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.591799 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:16.592128 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.592192 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.592476 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.594570 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.594713 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.596479 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.596559 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.596734 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:16.596891 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:16.596993 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:16.597075 [ 32111 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.597514 [ 32111 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.597537 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:16.597635 [ 31897 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:16.597781 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.597804 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:16.597806 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.597911 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:16.598032 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:16.598046 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.598694 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:16.600129 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.600243 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.600438 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:16.601261 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:16.601473 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:16.601877 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:16.601985 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.602039 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.602328 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:16.602424 [ 31884 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.602801 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.602854 [ 31884 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.603127 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:16.603168 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.603411 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.603805 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:16.603954 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:16.604063 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:16.605143 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.605234 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.606944 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:16.607135 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.607206 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.607562 [ 32106 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.607959 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.608049 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:16.607980 [ 32106 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.608387 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.608438 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:16.608762 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.609364 [ 31890 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:16.609466 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:16.609633 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:16.610313 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.610448 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.611915 [ 31890 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 542ms, ratio: 0.025641028, points per minute: 28.384901)
2026.05.06 08:49:16.611972 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.612094 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:16.612182 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:16.612312 [ 32100 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.612336 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:16.612603 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.612863 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.612880 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:16.612967 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:16.613067 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:16.613454 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:16.613554 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:16.613613 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:16.613719 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:16.614911 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.615020 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:16.615925 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:16.616099 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:16.616417 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:16.616513 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:16.616891 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:16.617884 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.618226 [ 31869 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.618337 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:16.618479 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:16.618570 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.618604 [ 31890 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:16.618795 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:16.618837 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.618939 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:16.620788 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.622679 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:16.622985 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:16.623072 [ 31881 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:16.624320 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:16.624617 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:16.624757 [ 31890 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:16.624854 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:16.624960 [ 31890 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:16.625407 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:16.625483 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:16.625621 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:16.625737 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:16.626358 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:16.627267 [ 31890 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:16.628191 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:16.629111 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:16.629239 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:16.629365 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:16.629881 [ 31890 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:16.630407 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:16.630604 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:16.631113 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:16.631146 [ 32334 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:16.631305 [ 32334 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:16.631676 [ 32334 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006851438 sec. (145.955 rows/sec., 3.42 KiB/sec.)
2026.05.06 08:49:16.631705 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:16.631809 [ 32334 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:16.632516 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:16.632760 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:16.633194 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:16.634806 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:16.634950 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:16.636181 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:16.636854 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:16.638072 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:16.638220 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:16.638369 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:16.638669 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.638883 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.639045 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:16.639051 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:16.639157 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:16.639385 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:16.640427 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:16.640593 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:16.640738 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:16.640998 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.641201 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.641363 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:16.641481 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:16.641578 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:16.641670 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:16.641705 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:16.641809 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:16.641942 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:16.642873 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:16.643018 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:16.643184 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:16.643441 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.643698 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.643873 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:16.643989 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:16.643988 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:16.644212 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:16.644707 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:16.645279 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:16.645518 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:16.645685 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:16.645984 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.646210 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.646388 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:16.646518 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:16.646733 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:16.646973 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:16.647087 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:16.647649 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:16.647998 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:16.648131 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:16.648186 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:16.648280 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:16.648560 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.648779 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.648935 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:16.649030 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:16.649227 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:16.649250 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:16.649595 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:16.650294 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:16.650425 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:16.650589 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:16.650858 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.651057 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.651207 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:16.651308 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:16.651546 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:16.652373 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:16.652525 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:16.652711 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:16.652996 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.653218 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:16.653293 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:16.653371 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:16.653419 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:16.653465 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:16.653569 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:16.653681 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:16.653826 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.654017 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.654153 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:16.654224 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:16.654234 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:16.654447 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:16.654973 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:16.655272 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:16.655385 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:16.655418 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:16.655526 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:16.655536 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:16.655773 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.655949 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.656091 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:16.656171 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:16.656334 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:16.657230 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:16.657337 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:16.657458 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:16.657417 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.657688 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.657857 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.658006 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:16.658091 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:16.658281 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:16.658771 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:16.659157 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:16.659266 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:16.659388 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:16.659648 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.659818 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.659947 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:16.660025 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:16.660184 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:16.660224 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.660639 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.660927 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.661100 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:16.661109 [ 32276 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:16.661213 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:16.661335 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:16.661578 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.661752 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.661883 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:16.661961 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:16.662126 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:16.662926 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:16.663030 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:16.663151 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:16.663367 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.663551 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.663705 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:16.663782 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:16.663944 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:16.664581 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:16.664681 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:16.664795 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:16.665007 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.665173 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:16.665295 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:16.665369 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:16.665544 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:16.665975 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:16.666529 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:16.666858 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:16.666933 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:16.670839 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:16.672276 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.672989 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:16.673357 [ 31859 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:17.253116 [ 31843 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:17.253680 [ 31843 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:17.253737 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:17.254089 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:17.254132 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:17.254482 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:17.257137 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:17.257229 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:17.259579 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:17.259728 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:17.259943 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:17.260108 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:17.260225 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:17.260289 [ 31840 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:17.260708 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:17.260782 [ 31840 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:17.261080 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:17.261089 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:17.261171 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:17.261212 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:17.261384 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:17.261575 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:17.261989 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:17.263104 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:17.263677 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:17.263706 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:17.264489 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:17.264749 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:17.265166 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:17.265181 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:17.265665 [ 31837 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:17.265764 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:17.266050 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:17.266375 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:17.266777 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:17.267243 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:17.267389 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:17.267542 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:17.268414 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:17.270386 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:17.270876 [ 31832 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:17.271335 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:17.271447 [ 31832 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:17.271794 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:17.271929 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:17.272240 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:17.272724 [ 31841 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:17.272821 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:17.272979 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:17.273619 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:17.273821 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:17.275158 [ 31841 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 645ms, ratio: 0.025641028, points per minute: 23.852118)
2026.05.06 08:49:17.275217 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:17.275296 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:17.275339 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:17.275437 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:17.276066 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:17.276149 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:17.276152 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:17.276266 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:17.276275 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:17.276363 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:17.276740 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:17.278191 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:17.278883 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:17.279062 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:17.279250 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:17.279390 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:17.279938 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:17.281124 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:17.281269 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:17.281370 [ 31841 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:17.281558 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:17.281710 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:17.285886 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:17.287327 [ 31841 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:17.287418 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:17.287547 [ 31841 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:17.288251 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:17.288326 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:17.288440 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:17.288567 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:17.290604 [ 31841 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:17.292845 [ 31841 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:17.293258 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:17.293435 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:17.293838 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:17.294368 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:17.295088 [ 31826 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:17.295208 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:17.295580 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:17.300107 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:17.300262 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:17.300409 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:17.300701 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.300910 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.300973 [ 31826 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:17.301068 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:17.301163 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:17.301366 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:17.302269 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:17.302394 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:17.302607 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:17.302875 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.303071 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.303224 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:17.303316 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:17.303521 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:17.304634 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:17.304762 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:17.304901 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:17.305146 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.305339 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.305491 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:17.305603 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:17.305803 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:17.306755 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:17.306877 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:17.307013 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:17.307260 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.307452 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.307661 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:17.307768 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:17.307963 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:17.309023 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:17.309150 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:17.309292 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:17.309555 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.309696 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:17.309757 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.309910 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:17.310005 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:17.310199 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:17.310692 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:17.311153 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:17.311272 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:17.311413 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:17.311712 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.311922 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.312071 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:17.312160 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:17.312350 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:17.313106 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:17.313226 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:17.313359 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:17.313626 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.313820 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:17.313964 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:17.314054 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:17.314240 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:17.314769 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:17.315458 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:17.315941 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:17.316035 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:17.320379 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:17.321700 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:17.322340 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:17.322847 [ 31823 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:17.323285 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:17.323353 [ 31823 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:17.323725 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:17.325581 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:17.327268 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:17.328104 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:17.328259 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:17.328373 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:17.331195 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:17.333614 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:17.333783 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:17.335338 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=9, jobs=9). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:17.335954 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=9, jobs=9). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:17.337771 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:17.340177 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:17.340270 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:17.340399 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:17.340531 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:17.345357 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:17.345570 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:17.346071 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:17.346689 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:17.347720 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:17.348125 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:17.352717 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:17.352872 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:17.353027 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:17.353315 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.353577 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.353755 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:17.353855 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:17.354063 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:17.355017 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:17.355147 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:17.355291 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:17.355568 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.355815 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.355976 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:17.356070 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:17.356268 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:17.357366 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:17.357511 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:17.357653 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:17.357910 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.358124 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.358284 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:17.358380 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:17.358589 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:17.359683 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:17.359826 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:17.359973 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:17.360242 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.360442 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.360619 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:17.360716 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:17.360909 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:17.362005 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:17.362136 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:17.362299 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:17.362574 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.362783 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.362940 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:17.363037 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:17.363238 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:17.364303 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:17.364435 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:17.364600 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:17.364855 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.365058 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.365211 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:17.365303 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:17.365494 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:17.366272 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:17.366393 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:17.366549 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:17.366799 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.366998 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:17.367147 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:17.367238 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:17.367430 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:17.368026 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:17.368731 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:17.369138 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:17.369230 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:17.373247 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:17.374582 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:17.375294 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:17.375788 [ 31821 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:17.649963 [ 31795 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1257ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:18.206474 [ 31784 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:18.206990 [ 31784 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:18.207024 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:18.207327 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:18.207440 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:18.207689 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:18.209987 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:18.210080 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:18.212024 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:18.212108 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:18.212288 [ 31812 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:18.212518 [ 31812 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:18.212640 [ 31812 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:18.212688 [ 31779 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:18.213037 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:18.213114 [ 31779 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:18.213333 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:18.213380 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:18.213390 [ 31812 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:18.213564 [ 31812 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:18.213679 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:18.213692 [ 31812 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:18.214295 [ 31812 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:18.215394 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:18.215653 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:18.215870 [ 31812 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:18.216615 [ 31812 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:18.216907 [ 31812 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:18.217298 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:18.217314 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:18.217342 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:18.217752 [ 31777 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:18.217876 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:18.218133 [ 31777 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:18.218135 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:18.218420 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:18.218460 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:18.218851 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:18.219561 [ 31812 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:18.219792 [ 31812 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:18.219919 [ 31812 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:18.220689 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:18.220737 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:18.222607 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:18.222630 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:18.223074 [ 31769 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:18.223653 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:18.223759 [ 31769 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:18.224071 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:18.224112 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:18.224383 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:18.225402 [ 31812 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:18.225527 [ 31812 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:18.225735 [ 31812 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:18.226180 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:18.226225 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:18.228238 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:18.228304 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:18.228346 [ 31812 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 936ms, ratio: 0.025641028, points per minute: 16.436556)
2026.05.06 08:49:18.228584 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:18.228703 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:18.229315 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:18.229375 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:18.229414 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:18.229486 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:18.229558 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:18.229608 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:18.230118 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:18.231706 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:18.232452 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:18.232468 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:18.232690 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:18.233045 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:18.233482 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:18.234807 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:18.234912 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:18.234958 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:18.235055 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:18.235169 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:18.238974 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:18.240579 [ 31768 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:18.240686 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:18.240835 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:18.241406 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:18.241481 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:18.241620 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:18.241736 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:18.243671 [ 31768 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:18.246706 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:18.246907 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:18.247380 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:18.247997 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:18.249007 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:18.249382 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:18.254371 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:18.254542 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:18.254694 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:18.254981 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.255196 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.255355 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:18.255452 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:18.255718 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:18.256672 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:18.256799 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:18.256940 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:18.257197 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.257396 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.257582 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:18.257675 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:18.257868 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:18.258949 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:18.259074 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:18.259214 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:18.259466 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.259745 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.259925 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:18.260023 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:18.260215 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:18.261213 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:18.261339 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:18.261478 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:18.261745 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.261960 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.262113 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:18.262206 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:18.262393 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:18.263485 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:18.263663 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:18.263809 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:18.264067 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.264302 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.264473 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:18.264590 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:18.264788 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:18.265825 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:18.265973 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:18.266117 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:18.266377 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.266594 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.266745 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:18.266835 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:18.267025 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:18.267926 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:18.268060 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:18.268200 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:18.268455 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.268676 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:18.268826 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:18.268917 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:18.269110 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:18.269668 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:18.270333 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:18.270754 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:18.270847 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:18.274971 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:18.276533 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:18.277235 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:18.277716 [ 31765 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:18.278064 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:18.278089 [ 31765 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:18.278405 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:18.280322 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:18.282126 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:18.282934 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:18.283045 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:18.283146 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:18.286196 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:18.288677 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:18.288849 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:18.292890 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:18.295128 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:18.295211 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:18.295336 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:18.295453 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:18.300361 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:18.300562 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:18.301137 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:18.301756 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:18.302819 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:18.303218 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:18.308042 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:18.308196 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:18.308370 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:18.308673 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.308890 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.309066 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:18.309163 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:18.309371 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:18.310307 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:18.310434 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:18.310593 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:18.310849 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.311047 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.311236 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:18.311328 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:18.311534 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:18.312704 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:18.312847 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:18.312993 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:18.313253 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.313450 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.313632 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:18.313729 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:18.313921 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:18.314968 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:18.315127 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:18.315270 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:18.315544 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.315810 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.315942 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:18.316033 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:18.316193 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:18.317107 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:18.317214 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:18.317335 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:18.317565 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.317739 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.317868 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:18.317949 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:18.318112 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:18.318924 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:18.319030 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:18.319150 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:18.319371 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.319547 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.319703 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:18.319780 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:18.319940 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:18.320575 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:18.320676 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:18.320790 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:18.320993 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.321155 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:18.321276 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:18.321351 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:18.321520 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:18.321965 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:18.322554 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:18.322893 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:18.322969 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:18.326697 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:18.328030 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:18.328705 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:18.329187 [ 31764 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:18.329431 [ 31764 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:18.335402 [ 31767 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:18.339325 [ 31767 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:18.344777 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:18.345433 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:18.361721 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:18.502097 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0917
8. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003afe2f18
9. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
10. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
11. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
12. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
13. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
14. ? @ 0x00007f6e2fbe4ac3
15. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:18.570981 [ 31737 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9377ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:18.768713 [ 32314 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:18.769271 [ 32314 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:18 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE732EA2E721B
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:18.769713 [ 32314 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:18.883813 [ 31729 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 2545ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:19.125854 [ 31721 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1475ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:19.203789 [ 31718 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 5451ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:19.425963 [ 31716 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:19.426209 [ 31716 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:19.426392 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:19.426392 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:19.426718 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:19.426764 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:19.428855 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:19.428972 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:19.430236 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:19.430309 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:19.430437 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:19.430600 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:19.430704 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:19.430786 [ 31711 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:19.431229 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:19.431268 [ 31711 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:19.431344 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:19.431448 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:19.431511 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:19.431576 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:19.431667 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:19.431686 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:19.432257 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:19.433116 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:19.433153 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:19.433563 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:19.434124 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:19.434339 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:19.434565 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:19.434657 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:19.434831 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:19.434921 [ 31707 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:19.435189 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:19.435264 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:19.435334 [ 31707 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:19.435531 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:19.435598 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:19.435772 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:19.436087 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:19.436219 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:19.436338 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:19.436708 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:19.436913 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:19.438249 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:19.438454 [ 31705 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:19.438641 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:19.438712 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:19.438981 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:19.440697 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:19.441622 [ 31706 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:19.441724 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:19.441887 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:19.442374 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:19.443133 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:19.443268 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:19.443399 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:19.444038 [ 31706 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1201ms, ratio: 0.025641028, points per minute: 12.809839)
2026.05.06 08:49:19.444222 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:19.444317 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:19.444822 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:19.444913 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:19.445011 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:19.445525 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:19.446244 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:19.446930 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:19.447648 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:19.447831 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:19.448139 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:19.448447 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:19.448541 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:19.448685 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:19.449745 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:19.449869 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:19.449985 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:19.452673 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:19.454879 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:19.454958 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:19.455102 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:19.455218 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:19.455301 [ 31706 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:19.455398 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:19.455532 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:19.458304 [ 31706 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:19.460107 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:19.460331 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:19.460818 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:19.461242 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:19.462379 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:19.462816 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:19.467596 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:19.467780 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:19.467936 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:19.468225 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.468442 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.468635 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:19.468735 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:19.468946 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:19.469902 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:19.470030 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:19.470175 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:19.470430 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.470650 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.470807 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:19.470902 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:19.471094 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:19.472267 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:19.472404 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:19.472570 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:19.472825 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.473038 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.473199 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:19.473295 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:19.473487 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:19.474633 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:19.474769 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:19.474917 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:19.475173 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.475373 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.475544 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:19.475670 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:19.475865 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:19.476953 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:19.477082 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:19.477227 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:19.477516 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.477763 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.477977 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:19.478090 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:19.478299 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:19.479222 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:19.479329 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:19.479451 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:19.479751 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.479923 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.480053 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:19.480130 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:19.480292 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:19.480942 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:19.481056 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:19.481172 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:19.481382 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.481559 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:19.481683 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:19.481760 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:19.481935 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:19.482442 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:19.483034 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:19.483391 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:19.483471 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:19.487016 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:19.488568 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:19.489273 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:19.489767 [ 31723 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:19.490147 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:19.490222 [ 31723 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:19.490472 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:19.492274 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:19.493876 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:19.494534 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:19.494595 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:19.494651 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:19.496068 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:19.497153 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:19.497250 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:19.499097 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:19.500206 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:19.500250 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:19.500318 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:19.500381 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:19.504099 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:19.504224 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:19.504675 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:19.505094 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:19.505977 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:19.506266 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:19.509245 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:19.509356 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:19.509471 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:19.509684 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.509834 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.509948 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:19.510017 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:19.510161 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:19.510815 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:19.510910 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:19.511007 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:19.511183 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.511322 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.511432 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:19.511510 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:19.511653 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:19.512431 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:19.512534 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:19.512632 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:19.512797 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.512932 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.513049 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:19.513115 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:19.513234 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:19.513922 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:19.514009 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:19.514099 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:19.514252 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.514372 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.514466 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:19.514534 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:19.514646 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:19.515318 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:19.515402 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:19.515495 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:19.515687 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.515823 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.515918 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:19.515976 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:19.516117 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:19.516758 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:19.516840 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:19.516931 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:19.517094 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.517221 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.517318 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:19.517375 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:19.517492 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:19.517984 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:19.518066 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:19.518157 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:19.518322 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.518464 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:19.518578 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:19.518641 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:19.518762 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:19.519125 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:19.519573 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:19.519873 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:19.519941 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:19.523552 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:19.524331 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:19.524738 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:19.525157 [ 31698 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:19.525359 [ 31698 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:19.533770 [ 31694 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:19.539865 [ 31694 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:19.549005 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:19.550070 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:19.572307 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:19.572448 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:19.572770 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006729149 sec. (148.607 rows/sec., 3.48 KiB/sec.)
2026.05.06 08:49:19.572858 [ 32456 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:19.577046 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:19.577696 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:19.584477 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:19.585178 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:19.596646 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:19.599685 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:19.600168 [ 32284 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:19.970459 [ 31898 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 6631ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:20.141998 [ 32071 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:20.142462 [ 32071 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:20.142476 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:20.142845 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:20.142875 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:20.143207 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:20.145364 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:20.145430 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:20.147102 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:20.147266 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:20.147302 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:20.147545 [ 32154 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:20.147533 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:20.147762 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:20.148812 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:20.149046 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:20.149189 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:20.149267 [ 32154 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:20.149421 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:20.149596 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:20.149647 [ 32179 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:20.149791 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:20.149846 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:20.150039 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:20.151136 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:20.151518 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:20.151640 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:20.151819 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:20.152033 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:20.152348 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:20.152806 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:20.153793 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:20.154047 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:20.154178 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:20.154268 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:20.154289 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:20.154535 [ 32130 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:20.154791 [ 32130 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:20.154872 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:20.154974 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:20.155178 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:20.155236 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:20.156379 [ 32179 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:20.157109 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:20.157185 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:20.158809 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:20.158870 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:20.159238 [ 32066 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:20.159453 [ 32076 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:20.159566 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:20.159729 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:20.159737 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:20.159794 [ 32066 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:20.160108 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:20.160153 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:20.160496 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:20.161995 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:20.162189 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:20.162209 [ 32076 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 704ms, ratio: 0.025641028, points per minute: 21.853148)
2026.05.06 08:49:20.162314 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:20.162394 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:20.162492 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:20.162885 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:20.163091 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:20.163180 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:20.163275 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:20.163655 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:20.163817 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:20.164228 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:20.164497 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:20.164627 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:20.164727 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:20.165335 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:20.166270 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:20.166478 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:20.166924 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:20.167412 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:20.167427 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:20.168869 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:20.169017 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:20.169132 [ 32076 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:20.169719 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:20.169861 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:20.173856 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:20.175413 [ 32076 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:20.175544 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:20.175700 [ 32076 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:20.176676 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:20.176756 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:20.176895 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:20.177012 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:20.178608 [ 32076 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:49:20.180709 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:20.180868 [ 32076 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:20.181108 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:20.182067 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:20.182268 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:20.182707 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:20.183318 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:20.184406 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:20.184847 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:20.189706 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:20.189854 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:20.190006 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:20.190289 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.190552 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.190717 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:20.190815 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:20.191025 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:20.192110 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:20.192252 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:20.192400 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:20.192688 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.192912 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.193125 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:20.193246 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:20.193474 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:20.194668 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:20.194801 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:20.194944 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:20.195205 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.195422 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.195677 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:20.195791 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:20.195993 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:20.197007 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:20.197133 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:20.197276 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:20.197565 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.197765 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.197922 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:20.198018 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:20.198206 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:20.199292 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:20.199421 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:20.199588 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:20.199963 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.200187 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.200351 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:20.200449 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:20.200669 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:20.201667 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:20.201809 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:20.201956 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:20.202215 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.202414 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.202581 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:20.202673 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:20.202864 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:20.203730 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:20.203870 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:20.204013 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:20.204273 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.204474 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:20.204645 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:20.204736 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:20.204928 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:20.205472 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:20.206167 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:20.206589 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:20.206682 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:20.210728 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:20.212094 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:20.212787 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:20.213220 [ 32070 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:20.213538 [ 32070 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:20.213967 [ 32070 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:20.213991 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:20.214410 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:20.216631 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:20.218540 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:20.219431 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:20.219569 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:20.219732 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:20.222559 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:20.224896 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:20.225054 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:20.229101 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:20.231408 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:20.231487 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:20.231654 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:20.231773 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:20.236744 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:20.236954 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:20.237538 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:20.238155 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:20.239647 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:20.240062 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:20.246054 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:20.246254 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:20.246450 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:20.246845 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.247118 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.247325 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:20.247451 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:20.247778 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:20.249022 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:20.249201 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:20.249384 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:20.249744 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.250021 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.250227 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:20.250350 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:20.250614 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:20.252044 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:20.252218 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:20.252396 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:20.252749 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.253003 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.253204 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:20.253324 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:20.253579 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:20.254807 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:20.254944 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:20.255116 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:20.255446 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.255771 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.255972 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:20.256094 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:20.256330 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:20.257710 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:20.257862 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:20.258036 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:20.258306 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.258537 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.258699 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:20.258797 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:20.259007 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:20.260202 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:20.260383 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:20.260578 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:20.260890 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.261134 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.261320 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:20.261414 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:20.261626 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:20.262475 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:20.262625 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:20.262766 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:20.263034 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.263239 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:20.263392 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:20.263487 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:20.263780 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:20.264404 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:20.265111 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:20.265538 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:20.265631 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:20.270314 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:20.271430 [ 32063 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1146ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:20.271685 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:20.272348 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:20.272736 [ 31873 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:20.272918 [ 31873 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:21.204196 [ 31867 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 6293ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:21.211570 [ 32102 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:21.212043 [ 32102 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:21.212107 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:21.212304 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:21.212531 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:21.212598 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:21.215102 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:21.215146 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:21.216726 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:21.216814 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:21.216907 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:21.216976 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:21.217207 [ 31895 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:21.217624 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:21.217677 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:21.217767 [ 31895 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:21.217702 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:21.217943 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:21.218028 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:21.218095 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:21.218211 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:21.218388 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:21.219525 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:21.219896 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:21.219940 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:21.220085 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:21.220201 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:21.220475 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:21.220859 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:21.221977 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:21.222071 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:21.222147 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:21.222233 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:21.222394 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:21.222572 [ 32171 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:21.222935 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:21.222961 [ 32171 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:21.223197 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:21.223272 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:21.223676 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:21.225027 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:21.225164 [ 31925 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:21.225235 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:21.225349 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:21.225426 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:21.226562 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:21.226939 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:21.227270 [ 31925 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1046ms, ratio: 0.025641028, points per minute: 14.708047)
2026.05.06 08:49:21.227390 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:21.227449 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:21.227743 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:21.227865 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:21.227905 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:21.227968 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:21.227974 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:21.228023 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:21.228332 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:21.229487 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:21.229676 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:21.230440 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:21.230536 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:21.230622 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:21.230671 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:21.230990 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:21.231379 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:21.232437 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:21.232763 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:21.232926 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:21.233054 [ 31925 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:21.235000 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:21.235077 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:21.235199 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:21.235314 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:21.238715 [ 31925 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:21.238829 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:21.238960 [ 31925 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:21.239793 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:21.239998 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:21.240357 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:21.240787 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:21.241901 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:21.242286 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:21.242756 [ 31925 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:21.247044 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:21.247193 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:21.247345 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:21.247710 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.247935 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.248099 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:21.248199 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:21.248410 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:21.249361 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:21.249492 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:21.249652 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:21.249911 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.250129 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.250287 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:21.250383 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:21.250595 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:21.251759 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:21.251894 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:21.252037 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:21.252288 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.252484 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.252658 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:21.252756 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:21.252950 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:21.254001 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:21.254136 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:21.254296 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:21.254572 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.254771 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.254928 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:21.255023 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:21.255216 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:21.256342 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:21.256474 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:21.256637 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:21.256893 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.257101 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.257255 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:21.257353 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:21.257567 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:21.258558 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:21.258686 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:21.258829 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:21.259085 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.259285 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.259439 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:21.259479 [ 32161 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:21.259544 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:21.259770 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:21.260536 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:21.260660 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:21.260798 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:21.261050 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.261245 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:21.261393 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:21.261483 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:21.261706 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:21.262243 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:21.262958 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:21.263364 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:21.263455 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:21.265494 [ 32161 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:21.266827 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:21.268308 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:21.269028 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:21.269483 [ 32105 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:21.269893 [ 32105 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:21.270005 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:21.270468 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:21.270750 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:21.271347 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:21.272754 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:21.274729 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:21.275736 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:21.275910 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:21.276067 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:21.278955 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:21.281271 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:21.281425 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:21.283726 [ 32377 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:21.283799 [ 32377 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:21.283961 [ 32377 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.004164536 sec. (240.123 rows/sec., 5.63 KiB/sec.)
2026.05.06 08:49:21.284007 [ 32377 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:21.285527 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:21.288207 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:21.288294 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:21.288309 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:21.288420 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:21.288560 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:21.289175 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:21.293472 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:21.293690 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:21.294200 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:21.294621 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:21.295640 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:21.296102 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:21.297271 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:21.297901 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:21.301734 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:21.301892 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:21.302090 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:21.302401 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.302643 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.302812 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:21.302966 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:21.303242 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:21.304352 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:21.304497 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:21.304666 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:21.304964 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.305215 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.305428 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:21.305560 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:21.305770 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:21.306993 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:21.307151 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:21.307299 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:21.307596 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.307853 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.308103 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:21.308249 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:21.308543 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:21.309834 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:21.310004 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:21.310188 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:21.310480 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.310787 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.310880 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:21.311003 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:21.311142 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:21.311391 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:21.313047 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:21.313272 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:21.313483 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:21.313871 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.314140 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.314360 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:21.314488 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:21.314527 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:21.314769 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:21.315024 [ 32274 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:21.316205 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:21.316347 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:21.316497 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:21.316790 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.317015 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.317209 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:21.317306 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:21.317525 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:21.318389 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:21.318544 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:21.318691 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:21.318958 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.319226 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:21.319400 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:21.319533 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:21.319765 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:21.320385 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:21.321140 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:21.321605 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:21.321729 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:21.325556 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:21.326997 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:21.327839 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:21.328422 [ 32166 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:21.328651 [ 32166 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:21.518554 [ 32146 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1247ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:21.605378 [ 32108 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 38032ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:21.788375 [ 32099 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:21.795029 [ 32099 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:21.801365 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:21.802129 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:21.821006 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:21.821314 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:21.830245 [ 32150 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 2947ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:22.084140 [ 32189 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:22.090384 [ 32189 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:22.099795 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:22.100879 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:22.124327 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.124939 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.193078 [ 32120 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.193388 [ 32120 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.193480 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:22.193608 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:22.193784 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.193818 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.196042 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.196059 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.197560 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.197592 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.197919 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:22.198110 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:22.198225 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:22.198289 [ 31880 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.198697 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:22.198770 [ 31880 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.198857 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:22.198988 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:22.199072 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.199125 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:22.199140 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:22.199484 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.199962 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:22.200525 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.201126 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.201273 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:22.201898 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:22.202024 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.202117 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:22.202489 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:22.202694 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.202997 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:22.203111 [ 32168 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.203636 [ 32168 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.203642 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:22.203923 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.203973 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:22.204310 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.204348 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:22.204493 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:22.204623 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:22.205280 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.205755 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.206529 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.206905 [ 32129 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.207173 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.207277 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:22.207685 [ 32080 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.208023 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.208087 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:22.208397 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.209471 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.209909 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.210407 [ 32172 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:22.210553 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:22.210741 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:22.210821 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:22.211344 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.211549 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:22.211690 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:22.211784 [ 32173 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.211819 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:22.212154 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:22.212496 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.213228 [ 32172 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 971ms, ratio: 0.025641028, points per minute: 15.844095)
2026.05.06 08:49:22.213416 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:22.213531 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:22.214120 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.214122 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:22.214254 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:22.214351 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:22.214540 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:22.214865 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:22.215648 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.216161 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:22.216812 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:22.216844 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:22.216959 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:22.217044 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:22.217363 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:22.217805 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:22.218902 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:22.219041 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:22.219148 [ 32172 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:22.220954 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:22.223300 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:22.223381 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:22.223539 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:22.223683 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:22.224899 [ 32172 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:22.225024 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:22.225163 [ 32172 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:22.228149 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:22.228269 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:22.228334 [ 32172 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:22.228634 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:22.229045 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:22.230184 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:22.230442 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:22.231636 [ 32172 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:22.232969 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:22.233057 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:22.233144 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:22.233303 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.233421 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.233519 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:22.233581 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:22.233711 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:22.234210 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:22.234282 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:22.234368 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:22.234515 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.234619 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.234701 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:22.234749 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:22.234851 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:22.235404 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:22.235470 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:22.235558 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:22.235701 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.235802 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.235885 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:22.235944 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:22.236048 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:22.236586 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:22.236654 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:22.236729 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:22.236852 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.236946 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.237028 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:22.237075 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:22.237165 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:22.237729 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:22.237796 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:22.237874 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:22.238006 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.238129 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.238226 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:22.238287 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:22.238400 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:22.239019 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:22.239102 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:22.239196 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:22.239356 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.239490 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.239634 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:22.239700 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:22.239805 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:22.240199 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:22.240263 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:22.240348 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:22.240493 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.240611 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.240688 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:22.240736 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:22.240842 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:22.241171 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:22.241601 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:22.241849 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:22.241913 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:22.245663 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:22.246332 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.246723 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.247102 [ 32153 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:22.247385 [ 32153 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.247740 [ 32153 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:22.247845 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:22.248260 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.250376 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.252268 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:22.253208 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:22.253326 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:22.253433 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:22.256573 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:22.259007 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:22.259166 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:22.263221 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:22.263314 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:22.263378 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:22.263467 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:22.263548 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:22.266933 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:22.267070 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:22.267523 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:22.268184 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:22.269229 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:22.269680 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:22.274872 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:22.275027 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:22.275184 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:22.275469 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.275735 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.275901 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:22.276000 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:22.276206 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:22.277154 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:22.277284 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:22.277426 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:22.277762 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.277987 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.278149 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:22.278245 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:22.278457 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:22.279585 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:22.279761 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:22.279929 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:22.280237 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.280497 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.280698 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:22.280800 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:22.280997 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:22.282081 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:22.282211 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:22.282358 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:22.282637 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.282838 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.283041 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:22.283168 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:22.283369 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:22.284600 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:22.284738 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:22.284888 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:22.285147 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.285361 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.285534 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:22.285634 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:22.285832 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:22.286823 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:22.286952 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:22.287097 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:22.287353 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.287580 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.287798 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:22.287925 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:22.288151 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:22.288959 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:22.289064 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:22.289182 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:22.289397 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.289580 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:22.289709 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:22.289787 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:22.289957 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:22.290547 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:22.291197 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:22.291698 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:22.291832 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:22.296421 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:22.297891 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.298629 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.299106 [ 32135 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:22.299322 [ 32135 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:22.490889 [ 32191 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.491329 [ 32191 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.491463 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:22.491733 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:22.491917 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.492129 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.495035 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.495107 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.497064 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.497135 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.497373 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:22.497546 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:22.497651 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:22.497670 [ 32175 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.497937 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:22.498027 [ 32175 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.498183 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.498320 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:22.498559 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:22.498607 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.498661 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:22.498789 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:22.499521 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:22.500018 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.500462 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.500596 [ 32313 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:22.501023 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:22.501136 [ 32313 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:22 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE733C8A26758
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:22.501566 [ 32313 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:22.501625 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:22.501716 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:22.501824 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.501998 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:22.502186 [ 32077 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.502246 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.502568 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:22.502589 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:22.502720 [ 32077 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.502926 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.503031 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:22.503328 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.503872 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:22.503944 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:22.503999 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:22.504610 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.505086 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.506278 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.506315 [ 32194 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:22.506372 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:22.506455 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:22.506685 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.507039 [ 32123 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.507288 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:22.507383 [ 32123 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.507575 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.507752 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:22.508001 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.508521 [ 32194 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 277ms, ratio: 0.025641028, points per minute: 55.54013)
2026.05.06 08:49:22.508624 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:22.508673 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:22.509004 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.509101 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:22.509149 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:22.509197 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:22.509433 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:22.509699 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.510429 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:22.510601 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:22.511049 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:22.511060 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:22.511130 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:22.511132 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:22.511199 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:22.511316 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:22.511383 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:22.511837 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:22.512525 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:22.513113 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:22.513269 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:22.513402 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:22.513588 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:22.513680 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:22.518033 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:22.518168 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:22.518227 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:22.518308 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:22.518385 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:22.519587 [ 32194 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:22.519728 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:22.519857 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:22.521860 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:22.521977 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:22.522131 [ 32194 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:22.522351 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:22.522861 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:22.523684 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:22.523950 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:22.524646 [ 32194 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:22.526526 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:22.526629 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:22.526731 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:22.526916 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.527060 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.527161 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:22.527222 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:22.527350 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:22.527983 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:22.528069 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:22.528162 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:22.528324 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.528449 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.528557 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:22.528617 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:22.528736 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:22.529434 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:22.529527 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:22.529626 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:22.529782 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.529904 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.530003 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:22.530063 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:22.530177 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:22.530815 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:22.530897 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:22.530986 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:22.531139 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.531256 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.531351 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:22.531411 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:22.531535 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:22.532222 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:22.532306 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:22.532399 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:22.532566 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.532692 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.532787 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:22.532848 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:22.532966 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:22.533600 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:22.533681 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:22.533772 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:22.533927 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.534049 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.534155 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:22.534212 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:22.534326 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:22.534825 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:22.534903 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:22.534991 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:22.535145 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.535264 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:22.535354 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:22.535413 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:22.535538 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:22.535919 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:22.536365 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:22.536643 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:22.536704 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:22.539519 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:22.540297 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.540692 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.541026 [ 32038 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:22.541329 [ 32038 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:22.541640 [ 32038 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:22.541713 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:22.542056 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:22.543699 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:22.545965 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:22.546893 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:22.547005 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:22.547122 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:22.549968 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:22.552256 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:22.552401 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:22.556346 [ 32053 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9883ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:22.556422 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:22.559010 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:22.559106 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:22.559235 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:22.559357 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:22.564666 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:22.564911 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:22.565535 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:22.566166 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:22.567463 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:22.567986 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:22.573403 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:22.573586 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:22.573746 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:22.574041 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.574276 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.574443 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:22.574560 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:22.574776 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:22.575806 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:22.575948 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:22.576099 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:22.576382 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.576609 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.576772 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:22.576871 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:22.577081 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:22.578229 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:22.578394 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:22.578556 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:22.578826 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.579031 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.579194 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:22.579294 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:22.579515 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:22.580600 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:22.580736 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:22.580884 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:22.581154 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.581361 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.581538 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:22.581638 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:22.581841 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:22.583006 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:22.583141 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:22.583329 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:22.583703 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.583978 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.584183 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:22.584306 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:22.584573 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:22.585825 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:22.585983 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:22.586163 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:22.586470 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.586736 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.586925 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:22.587037 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:22.587283 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:22.588356 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:22.588539 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:22.588715 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:22.589024 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.589286 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:22.589532 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:22.589659 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:22.589902 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:22.590595 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:22.591332 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:22.591784 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:22.591873 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:22.595863 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:22.597406 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.598178 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.598601 [ 32048 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:22.598734 [ 32048 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:22.882577 [ 32039 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:22.886670 [ 32039 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:22.893425 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:22.894191 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:22.909252 [ 32474 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:22.909406 [ 32474 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:22.909758 [ 32474 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.005303157 sec. (188.567 rows/sec., 4.42 KiB/sec.)
2026.05.06 08:49:22.909877 [ 32474 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:22.913249 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:22.913694 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:22.917005 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:22.917536 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:22.924215 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.925945 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:22.926247 [ 32274 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:23.437353 [ 32028 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:23.437865 [ 32028 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:23.437955 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:23.438300 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:23.438349 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:23.438816 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:23.440893 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:23.440957 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:23.442649 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:23.442735 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:23.442936 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:23.443108 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:23.443222 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:23.443944 [ 32025 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:23.444061 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:23.444181 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:23.444292 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:23.444363 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:23.444387 [ 32025 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:23.444609 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:23.444735 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:23.444882 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:23.444998 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:23.446047 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:23.446068 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:23.446704 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:23.446849 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:23.447038 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:23.447356 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:23.447757 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:23.448694 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:23.449147 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:23.449281 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:23.449393 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:23.449421 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:23.449777 [ 32022 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:23.450137 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:23.450196 [ 32022 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:23.450451 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:23.450451 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:23.450762 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:23.452286 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:23.452544 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:23.454273 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:23.454680 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:23.454814 [ 32029 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:23.454920 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:23.454972 [ 32049 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:23.455048 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:23.455324 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:23.455352 [ 32049 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:23.455491 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:23.455695 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:23.455957 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:23.457277 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:23.457650 [ 32029 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 933ms, ratio: 0.025641028, points per minute: 16.489407)
2026.05.06 08:49:23.457716 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:23.457826 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:23.457923 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:23.458533 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:23.458654 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:23.458763 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:23.459004 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:23.459330 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:23.459390 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:23.459783 [ 31998 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:23.459833 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:23.459943 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:23.460045 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:23.460131 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:23.460466 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:23.461047 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:23.461690 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:23.461878 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:23.462216 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:23.462233 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:23.462726 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:23.462828 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:23.464223 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:23.464439 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:23.464586 [ 32029 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:23.464933 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:23.465175 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:23.465326 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:23.469297 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:23.471009 [ 32029 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:23.471122 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:23.471255 [ 32029 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:23.471431 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:23.471530 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:23.471685 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:23.471808 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:23.474072 [ 32029 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:49:23.476249 [ 32193 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1958ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:23.476675 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:23.476816 [ 32029 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:23.476868 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:23.477295 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:23.477851 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:23.478765 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:23.479197 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:23.484062 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:23.484219 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:23.484377 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:23.484691 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.484931 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.485098 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:23.485198 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:23.485413 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:23.486362 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:23.486492 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:23.486658 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:23.486924 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.487151 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.487318 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:23.487414 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:23.487703 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:23.488808 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:23.488954 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:23.489101 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:23.489396 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.489624 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.489790 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:23.489886 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:23.490083 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:23.491086 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:23.491212 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:23.491360 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:23.491707 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.491941 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.492107 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:23.492204 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:23.492403 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:23.493574 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:23.493708 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:23.493869 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:23.494181 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.494423 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.494633 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:23.494737 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:23.494947 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:23.495988 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:23.496120 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:23.496271 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:23.496586 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.496796 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.496955 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:23.497050 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:23.497262 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:23.498190 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:23.498325 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:23.498473 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:23.498762 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.498971 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:23.499134 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:23.499246 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:23.499462 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:23.500071 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:23.500767 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:23.501238 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:23.501354 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:23.507552 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:23.508753 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:23.509307 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:23.509699 [ 31996 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:23.509897 [ 31996 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:23.510102 [ 31996 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:23.510207 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:23.510591 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:23.512718 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:23.514834 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:23.515819 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:23.515943 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:23.516048 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:23.518889 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:23.521206 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:23.521361 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:23.525479 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:23.527954 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:23.528039 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:23.528164 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:23.528283 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:23.533028 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:23.533217 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:23.533819 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:23.534397 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:23.535624 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:23.536019 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:23.540571 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:23.540720 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:23.540874 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:23.541155 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.541385 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.541559 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:23.541656 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:23.541862 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:23.542804 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:23.542930 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:23.543073 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:23.543331 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.543550 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.543750 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:23.543846 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:23.544040 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:23.545122 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:23.545261 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:23.545402 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:23.545672 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.545869 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.546024 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:23.546118 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:23.546306 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:23.547346 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:23.547479 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:23.547671 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:23.547931 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.548130 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.548289 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:23.548384 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:23.548593 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:23.549689 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:23.549816 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:23.549958 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:23.550213 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.550418 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.550585 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:23.550682 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:23.550877 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:23.551880 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:23.552009 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:23.552151 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:23.552405 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.552621 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.552773 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:23.552864 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:23.553071 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:23.553830 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:23.553951 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:23.554088 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:23.554334 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.554546 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:23.554693 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:23.554783 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:23.554975 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:23.555517 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:23.556200 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:23.556620 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:23.556712 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:23.561102 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:23.562425 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:23.563078 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:23.563577 [ 31989 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:23.563878 [ 31989 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:23.877592 [ 31973 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:23.883769 [ 31973 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:23.893011 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:23.894057 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:23.916099 [ 32439 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:23.916231 [ 32439 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:23.916554 [ 32439 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006390898 sec. (156.473 rows/sec., 3.67 KiB/sec.)
2026.05.06 08:49:23.916636 [ 32439 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:23.920895 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:23.921491 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:23.928251 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:23.928920 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:23.940355 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:23.943268 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:23.943768 [ 32284 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:24.621567 [ 31960 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:24.622011 [ 31960 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:24.622050 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:24.622334 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:24.622426 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:24.622635 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:24.625264 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:24.625329 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:24.627049 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:24.627106 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:24.627330 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:24.627530 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:24.627691 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:24.627697 [ 31957 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:24.628155 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:24.628167 [ 31957 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:24.628494 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:24.628524 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:24.628554 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:24.628663 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:24.628788 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:24.628850 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:24.629391 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:24.630362 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:24.630562 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:24.630747 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:24.631421 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:24.631703 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:24.631985 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:24.632087 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:24.632268 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:24.632600 [ 31950 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:24.632593 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:24.632931 [ 31950 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:24.632991 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:24.633208 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:24.633359 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:24.633484 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:24.634078 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:24.634213 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:24.634326 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:24.635269 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:24.635330 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:24.637057 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:24.637144 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:24.637420 [ 31972 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:24.637768 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:24.637847 [ 31972 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:24.638151 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:24.638180 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:24.638448 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:24.639495 [ 31956 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:24.639643 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:24.639802 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:24.640220 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:24.640315 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:24.641808 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:24.641998 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:24.642079 [ 31956 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1166ms, ratio: 0.025641028, points per minute: 13.194354)
2026.05.06 08:49:24.642254 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:24.642341 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:24.642521 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:24.642590 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:24.642646 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:24.642969 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:24.643060 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:24.643153 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:24.643663 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:24.645121 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:24.645291 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:24.645825 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:24.646014 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:24.646341 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:24.646782 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:24.647794 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:24.647952 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:24.647947 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:24.648075 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:24.648176 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:24.652149 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:24.652191 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:24.652228 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:24.652289 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:24.652364 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:24.653614 [ 31956 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:24.653705 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:24.653809 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:24.655318 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:24.655410 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:24.655778 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:24.656253 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:24.657093 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:24.657297 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:24.657297 [ 31956 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:24.659249 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:24.659330 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:24.659414 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:24.659574 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.659704 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.659787 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:24.659838 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:24.659943 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:24.660409 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:24.660471 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:24.660554 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:24.660683 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.660781 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.660857 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:24.660902 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:24.660994 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:24.661524 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:24.661587 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:24.661660 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:24.661785 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.661890 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.661970 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:24.662024 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:24.662113 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:24.662602 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:24.662663 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:24.662732 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:24.662848 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.662939 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.663011 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:24.663056 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:24.663143 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:24.663703 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:24.663776 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:24.663867 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:24.664104 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.664292 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.664467 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:24.664582 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:24.664746 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:24.665569 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:24.665674 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:24.665792 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:24.665999 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.666160 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.666283 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:24.666356 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:24.666528 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:24.667124 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:24.667221 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:24.667328 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:24.667542 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.667750 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:24.667873 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:24.667968 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:24.668120 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:24.668563 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:24.669074 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:24.669399 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:24.669474 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:24.673735 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:24.674814 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:24.675328 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:24.675726 [ 31944 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:24.675983 [ 31944 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:24.676291 [ 31944 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:24.676341 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:24.676675 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:24.678898 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:24.680903 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:24.681788 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:24.681901 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:24.682002 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:24.684989 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:24.687449 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:24.687634 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:24.691889 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:24.694346 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:24.694425 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:24.694563 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:24.694679 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:24.699674 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:24.699871 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:24.700405 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:24.701032 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:24.702221 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:24.702628 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:24.707498 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:24.707705 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:24.707863 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:24.708150 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.708387 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.708571 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:24.708671 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:24.708883 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:24.709883 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:24.710018 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:24.710168 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:24.710439 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.710668 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.710834 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:24.710954 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:24.711157 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:24.712342 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:24.712497 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:24.712663 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:24.712932 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.713141 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.713305 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:24.713405 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:24.713640 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:24.714692 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:24.714825 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:24.714975 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:24.715244 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.715457 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.715673 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:24.715821 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:24.716065 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:24.717243 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:24.717380 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:24.717550 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:24.717828 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.718050 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.718234 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:24.718336 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:24.718560 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:24.719697 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:24.719842 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:24.719998 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:24.720274 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.720485 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.720662 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:24.720757 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:24.720963 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:24.721814 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:24.721944 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:24.722090 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:24.722356 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.722602 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:24.722762 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:24.722860 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:24.723069 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:24.723686 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:24.724414 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:24.724860 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:24.724956 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:24.729124 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:24.730585 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:24.731285 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:24.731869 [ 31940 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:24.732076 [ 31940 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:24.866098 [ 31934 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:24.872318 [ 31934 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:24.881540 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:24.882618 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:24.904950 [ 32469 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:24.905091 [ 32469 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:24.905387 [ 32469 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006770318 sec. (147.704 rows/sec., 3.46 KiB/sec.)
2026.05.06 08:49:24.905525 [ 32469 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:24.909462 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:24.910114 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:24.916839 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:24.917554 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:24.929551 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:24.932426 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:24.932923 [ 32277 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:25.067584 [ 32128 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.068050 [ 32128 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.068105 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:25.068453 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.068496 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.068906 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.070926 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.071001 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.072718 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.072895 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.073075 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:25.073283 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:25.073398 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:25.073489 [ 31919 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.073895 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:25.073963 [ 31919 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.074073 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:25.074190 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:25.074207 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.074259 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.074298 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:25.074596 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.074896 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:25.075909 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.076386 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:25.076427 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.077153 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:25.077342 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:25.077535 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.077776 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:25.077956 [ 31930 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.078309 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:25.078319 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:25.078641 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.079495 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.079935 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:25.080082 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:25.080197 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:25.080642 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.082366 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.082818 [ 31916 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.083094 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.083192 [ 31916 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.083366 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.083459 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:25.083778 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.085002 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.085525 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.085530 [ 31921 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:25.085709 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:25.085840 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:25.086490 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.086847 [ 31909 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.087240 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.087562 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.087639 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:25.088418 [ 31921 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 431ms, ratio: 0.025641028, points per minute: 35.695168)
2026.05.06 08:49:25.088631 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:25.088637 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:25.088734 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:25.088786 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:25.088892 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:25.089325 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:25.089423 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:25.089464 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.089536 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:25.090035 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:25.091324 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.091480 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:25.091781 [ 31905 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.091932 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:25.092176 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.092455 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:25.092475 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.092658 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:25.092995 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:25.093441 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:25.094174 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:25.094321 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:25.094868 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:25.094944 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.095014 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:25.095126 [ 31921 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:25.096371 [ 31640 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #5
2026.05.06 08:49:25.096396 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:49:25.096637 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:25.096788 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.098352 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:25.100429 [ 31921 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:25.100551 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:25.100676 [ 31921 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:25.100838 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:25.100938 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:25.101062 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:25.101180 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:25.103065 [ 31921 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:25.105255 [ 31921 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:25.106049 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:25.106238 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:25.106707 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:25.107226 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:25.108207 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:25.108635 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:25.113532 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:25.113683 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:25.113835 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:25.114119 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.114335 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.114496 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:25.114614 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:25.114821 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:25.115815 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:25.115950 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:25.116094 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:25.116352 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.116575 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.116730 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:25.116822 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:25.117014 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:25.118160 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:25.118292 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:25.118433 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:25.118706 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.118902 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.119060 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:25.119170 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:25.119364 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:25.120390 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:25.120539 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:25.120681 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:25.120930 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.121124 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.121276 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:25.121371 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:25.121575 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:25.122697 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:25.122827 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:25.122970 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:25.123224 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.123426 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.123640 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:25.123739 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:25.123962 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:25.125048 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:25.125178 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:25.125328 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:25.125660 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.125907 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.126112 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:25.126210 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:25.126407 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:25.127242 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:25.127368 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:25.127526 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:25.127811 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.128026 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:25.128172 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:25.128262 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:25.128453 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:25.128995 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:25.129734 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:25.130148 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:25.130239 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:25.134187 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:25.135498 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:25.136197 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:25.136713 [ 31903 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:25.137150 [ 31903 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.137528 [ 31903 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:25.137598 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.137936 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.139710 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.141434 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:25.142427 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:25.142557 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:25.142662 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:25.145455 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:25.147837 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:25.148017 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:25.152203 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:25.154465 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:25.154562 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:25.154685 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:25.154802 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:25.159810 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:25.160016 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:25.160593 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:25.161120 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:25.162255 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:25.162688 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:25.167494 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:25.167696 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:25.167854 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:25.168189 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.168431 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.168616 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:25.168718 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:25.168930 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:25.169940 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:25.170071 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:25.170218 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:25.170478 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.170707 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.170863 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:25.170957 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:25.171154 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:25.172292 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:25.172425 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:25.172586 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:25.172838 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.173034 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.173193 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:25.173320 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:25.173528 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:25.174517 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:25.174646 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:25.174787 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:25.175034 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.175248 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.175399 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:25.175492 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:25.175734 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:25.176826 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:25.176955 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:25.177101 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:25.177361 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.177585 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.177757 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:25.177873 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:25.178070 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:25.179095 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:25.179223 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:25.179380 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:25.179700 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.179904 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.180055 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:25.180145 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:25.180337 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:25.181104 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:25.181226 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:25.181362 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:25.181629 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.181842 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:25.181989 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:25.182082 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:25.182272 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:25.182817 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:25.183464 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:25.183989 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:25.184086 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:25.188341 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:25.189819 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:25.190526 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:25.190987 [ 31901 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:25.191236 [ 31901 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:25.341807 [ 31891 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1865ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:25.512997 [ 31897 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:25.518986 [ 31897 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:25.528053 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:25.529092 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:25.554371 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:25.554912 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:25.792552 [ 31876 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 7221ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:25.892339 [ 32100 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 3336ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:25.917813 [ 31869 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.918194 [ 31869 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.918322 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:25.918562 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.918765 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.918921 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.921645 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.921847 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.923762 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.923886 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.924007 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:25.924151 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:25.924275 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:25.924273 [ 31875 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.925385 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:25.925620 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:25.925799 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:25.926144 [ 31875 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.926142 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:25.926408 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:25.926460 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.926530 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.926839 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.927873 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:25.928109 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.928583 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.928562 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:25.928827 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:25.929186 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:25.929676 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:25.930889 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.931035 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:25.931166 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:25.931202 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.931277 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:25.931732 [ 31888 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.932142 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:25.932161 [ 31888 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.932415 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.932441 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.932737 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.934111 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.934340 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.935935 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.935961 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.936371 [ 31868 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.936421 [ 31881 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:25.936559 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:25.936717 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:25.936729 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.936782 [ 31868 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.937080 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:25.937150 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.937341 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.939098 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.939148 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.939238 [ 31881 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 834ms, ratio: 0.025641028, points per minute: 18.446783)
2026.05.06 08:49:25.939410 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:25.939495 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:25.940112 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:25.940200 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:25.940289 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:25.940757 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:25.940837 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.940933 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:25.941274 [ 32087 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.941643 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.941721 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:25.941844 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:25.941926 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.941970 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:25.942048 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:25.942706 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:25.942895 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:25.943198 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:25.943573 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:25.944388 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.944837 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:25.944956 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:25.945057 [ 31881 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:25.945109 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:25.946205 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:25.947705 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:25.947873 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:25.950276 [ 31881 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:25.950368 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:25.950471 [ 31881 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:25.952925 [ 31881 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:25.954254 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:25.955257 [ 31881 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:25.956739 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:25.956825 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:25.956949 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:25.957083 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:25.962108 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:25.962321 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:25.962860 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:25.963428 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:25.964584 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:25.965061 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:25.970145 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:25.970299 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:25.970466 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:25.970784 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.971005 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.971178 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:25.971280 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:25.971487 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:25.972490 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:25.972667 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:25.972813 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:25.973107 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.973310 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.973467 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:25.973583 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:25.973798 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:25.974954 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:25.975087 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:25.975229 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:25.975490 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.975747 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.975925 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:25.976049 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:25.976247 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:25.977436 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:25.977653 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:25.977816 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:25.978082 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.978280 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.978436 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:25.978549 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:25.978738 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:25.979858 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:25.979991 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:25.980136 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:25.980394 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.980625 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.980782 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:25.980878 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:25.981077 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:25.982063 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:25.982189 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:25.982332 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:25.982600 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.982798 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.982948 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:25.983045 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:25.983235 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:25.984020 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:25.984143 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:25.984278 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:25.984543 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.984741 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:25.984895 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:25.984999 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:25.985189 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:25.985726 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:25.986396 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:25.986817 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:25.986918 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:25.991127 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:25.992485 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:25.993152 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:25.993567 [ 31860 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:25.993851 [ 31860 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:25.994199 [ 31860 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:25.994247 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:25.994608 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:25.996639 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:25.998522 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:25.999375 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:25.999485 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:25.999634 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:26.003288 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:26.006382 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:26.006609 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:26.011044 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:26.013469 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:26.013574 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:26.013709 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:26.013844 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:26.018776 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:26.018972 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:26.019567 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:26.020175 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:26.021409 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:26.021837 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:26.026589 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:26.026745 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:26.026925 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:26.027211 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.027431 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.027648 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:26.027750 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:26.027978 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:26.028937 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:26.029069 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:26.029214 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:26.029494 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.029720 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.029876 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:26.029971 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:26.030165 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:26.031263 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:26.031393 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:26.031557 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:26.031856 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.032101 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.032285 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:26.032384 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:26.032610 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:26.033621 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:26.033767 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:26.033921 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:26.034202 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.034403 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.034571 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:26.034666 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:26.034856 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:26.036004 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:26.036138 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:26.036300 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:26.036590 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.036799 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.036956 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:26.037052 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:26.037251 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:26.038267 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:26.038397 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:26.038557 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:26.038813 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.039011 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.039163 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:26.039254 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:26.039446 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:26.040298 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:26.040424 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:26.040581 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:26.040847 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.041051 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:26.041197 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:26.041301 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:26.041493 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:26.042044 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:26.042722 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:26.043134 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:26.043226 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:26.047331 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:26.048943 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:26.049781 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:26.050297 [ 31857 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:26.050535 [ 31857 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:26.099696 [ 32086 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:26.105629 [ 32086 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:26.114902 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:26.115985 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:26.138821 [ 32455 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:26.138974 [ 32455 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:26.139290 [ 32455 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006854928 sec. (145.880 rows/sec., 3.42 KiB/sec.)
2026.05.06 08:49:26.139414 [ 32455 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:26.143762 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:26.144379 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:26.151403 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:26.152157 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:26.164953 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:26.167924 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:26.168400 [ 32281 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:26.404157 [ 31853 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:26.404684 [ 31853 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:26.404760 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:26.405549 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:26.405584 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:26.405959 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:26.407988 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:26.408306 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:26.409906 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:26.409983 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:26.410121 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:26.410268 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:26.410369 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:26.410458 [ 31847 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:26.410927 [ 31847 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:26.410926 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:26.411023 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:26.411128 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:26.411233 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:26.411287 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:26.411302 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:26.411589 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:26.411866 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:26.413966 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:26.414377 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:26.414430 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:26.415069 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:26.415262 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:26.415525 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:26.415682 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:26.415851 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:26.416862 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:26.417143 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:26.417283 [ 31843 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:26.417319 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:26.417454 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:26.417652 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:26.417710 [ 31843 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:26.417974 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:26.417982 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:26.418282 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:26.419866 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:26.419939 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:26.421461 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:26.421533 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:26.421994 [ 31840 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:26.422313 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:26.422328 [ 31840 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:26.422610 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:26.422659 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:26.422662 [ 31849 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:26.422873 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:26.422995 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:26.423035 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:26.424351 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:26.424640 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:26.425258 [ 31849 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 470ms, ratio: 0.025641028, points per minute: 32.733223)
2026.05.06 08:49:26.425427 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:26.425528 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:26.425883 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:26.426059 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:26.426142 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:26.426225 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:26.426297 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:26.426649 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:26.426719 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:26.426731 [ 31838 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:26.426763 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:26.426917 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:26.427067 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:26.427337 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:26.428048 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:26.428899 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:26.429127 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:26.429150 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:26.429415 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:26.429473 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:26.429856 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:26.431089 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:26.431215 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:26.431316 [ 31849 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:26.431749 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:26.431830 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:26.432035 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:26.432128 [ 31829 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:26.432596 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:26.432916 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:26.434934 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:26.436754 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:26.437148 [ 31828 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:26.437545 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:26.437900 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:26.438157 [ 31849 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:26.438256 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:26.438366 [ 31849 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:26.439311 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:26.439713 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:26.440568 [ 31849 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:26.441564 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:26.441825 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:26.441923 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:26.442073 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:26.442215 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:26.442319 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:26.442433 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:26.442457 [ 31849 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:26.442554 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:26.445357 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:26.447235 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:26.447421 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:26.447655 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:26.447797 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:26.447897 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:26.448354 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:26.449296 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:26.450368 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:26.451837 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:26.454038 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:26.454125 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:26.454253 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:26.454373 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:26.455098 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:26.455243 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:26.455395 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:26.455721 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.455940 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.456102 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:26.456199 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:26.456407 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:26.457346 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:26.457472 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:26.457628 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:26.457882 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.458100 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.458254 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:26.458349 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:26.458561 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:26.459255 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:26.459451 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:26.459665 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:26.459799 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:26.459941 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:26.460037 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:26.460195 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.460393 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.460455 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:26.460560 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:26.460657 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:26.460860 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:26.461397 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:26.461857 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:26.461947 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:26.461985 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:26.462126 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:26.462372 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.462580 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.462734 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:26.462825 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:26.463013 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:26.464113 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:26.464242 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:26.464383 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:26.464651 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.464855 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.465020 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:26.465115 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:26.465307 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:26.466279 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:26.466401 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:26.466555 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:26.466754 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:26.466806 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.466912 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:26.467001 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.467074 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:26.467151 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:26.467239 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:26.467372 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.467427 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:26.467645 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.467820 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:26.467923 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:26.468141 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:26.468227 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:26.468347 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:26.468482 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:26.468747 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.468965 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:26.469125 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:26.469131 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:26.469280 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:26.469282 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:26.469440 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:26.469532 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:26.469730 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.469944 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.470067 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:26.470107 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:26.470207 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:26.470427 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:26.470787 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:26.471189 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:26.471280 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:26.471602 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:26.471774 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:26.471920 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:26.472183 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.472386 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.472582 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:26.472683 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:26.472880 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:26.473905 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:26.474053 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:26.474200 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:26.474461 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.474685 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.474845 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:26.474940 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:26.474996 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:26.475156 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:26.476322 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:26.476354 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:26.476457 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:26.476627 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:26.476938 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.477021 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:26.477198 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.477369 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:26.477470 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:26.477481 [ 31825 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:26.477710 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:26.478735 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:26.478866 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:26.479013 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:26.479272 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.479473 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.479687 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:26.479783 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:26.479979 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:26.480758 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:26.480881 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:26.481020 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:26.481273 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.481489 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:26.481674 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:26.481769 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:26.481965 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:26.482533 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:26.483211 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:26.483680 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:26.483781 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:26.487679 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:26.489052 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:26.489703 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:26.490111 [ 31826 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:26.759672 [ 31795 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1418ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:26.805954 [ 32312 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:26.806467 [ 32312 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:26 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE734C93EAD02
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:26.806871 [ 32312 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:26.835693 [ 31796 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:26.841634 [ 31796 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:26.850876 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:26.851955 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:26.877330 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:26.877909 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:27.040798 [ 31791 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:27.041226 [ 31791 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:27.041298 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:27.041471 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:27.041657 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:27.041684 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:27.044323 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:27.044375 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:27.046158 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:27.046207 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:27.046403 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:27.046563 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:27.046666 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:27.046696 [ 31803 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:27.047049 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:27.047138 [ 31803 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:27.047326 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:27.047431 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:27.047446 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:27.047449 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:27.047593 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:27.047773 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:27.048193 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:27.049376 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:27.049732 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:27.049854 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:27.050448 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:27.050670 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:27.051007 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:27.051234 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:27.051459 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:27.052529 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:27.052859 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:27.052871 [ 31789 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:27.052987 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:27.053097 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:27.053264 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:27.053274 [ 31789 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:27.053624 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:27.053710 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:27.054085 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:27.055449 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:27.055994 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:27.057124 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:27.057744 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:27.058144 [ 31816 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:27.058576 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:27.058583 [ 31786 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:27.058762 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:27.058867 [ 31816 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:27.058945 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:27.059154 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:27.059295 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:27.059756 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:27.061066 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:27.061350 [ 31786 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 619ms, ratio: 0.025641028, points per minute: 24.853985)
2026.05.06 08:49:27.061439 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:27.061548 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:27.061647 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:27.062187 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:27.062280 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:27.062377 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:27.062704 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:27.062878 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:27.063137 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:27.063948 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:27.064061 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:27.064166 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:27.064274 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:27.064955 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:27.065136 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:27.065476 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:27.065957 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:27.066934 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:27.067185 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:27.067349 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:27.067458 [ 31786 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:27.069215 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:27.069360 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:27.072879 [ 31786 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:27.072980 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:27.073097 [ 31786 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:27.075778 [ 31786 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:27.076406 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:27.078329 [ 31786 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:27.078698 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:27.078782 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:27.078909 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:27.079027 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:27.084002 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:27.084194 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:27.084730 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:27.085314 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:27.086464 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:27.086889 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:27.091479 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:27.091681 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:27.091841 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:27.092131 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.092345 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.092525 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:27.092625 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:27.092834 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:27.093798 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:27.093932 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:27.094075 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:27.094334 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.094553 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.094709 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:27.094820 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:27.095016 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:27.096165 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:27.096300 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:27.096442 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:27.096715 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.096914 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.097073 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:27.097168 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:27.097358 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:27.098349 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:27.098476 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:27.098635 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:27.098907 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.099102 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.099268 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:27.099362 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:27.099592 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:27.100708 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:27.100838 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:27.100983 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:27.101237 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.101440 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.101609 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:27.101706 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:27.101902 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:27.102880 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:27.103004 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:27.103146 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:27.103399 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.103669 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.103825 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:27.103916 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:27.104109 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:27.104880 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:27.105036 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:27.105174 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:27.105426 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.105642 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:27.105791 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:27.105882 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:27.106075 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:27.106623 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:27.107306 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:27.107779 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:27.107875 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:27.111735 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:27.113049 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:27.113707 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:27.114265 [ 31774 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:27.114725 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:27.114721 [ 31774 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:27.115114 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:27.116796 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:27.118375 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:27.119119 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:27.119230 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:27.119331 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:27.122281 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:27.124607 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:27.124757 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:27.128781 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:27.130999 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:27.131078 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:27.131201 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:27.131317 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:27.136321 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:27.136524 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:27.137016 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:27.137589 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:27.138692 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:27.139078 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:27.143599 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:27.143778 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:27.143931 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:27.144214 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.144427 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.144604 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:27.144702 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:27.144911 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:27.145880 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:27.146006 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:27.146146 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:27.146400 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.146613 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.146766 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:27.146857 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:27.147061 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:27.148162 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:27.148286 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:27.148424 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:27.148695 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.148889 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.149042 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:27.149136 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:27.149370 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:27.150560 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:27.150711 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:27.150869 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:27.151152 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.151373 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.151583 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:27.151880 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:27.152089 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:27.153344 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:27.153499 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:27.153696 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:27.153976 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.154183 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.154338 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:27.154434 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:27.154645 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:27.155614 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:27.155783 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:27.155925 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:27.156176 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.156386 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.156550 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:27.156639 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:27.156829 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:27.157574 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:27.157691 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:27.157824 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:27.158069 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.158263 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:27.158408 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:27.158496 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:27.158703 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:27.159212 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:27.159896 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:27.160285 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:27.160385 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:27.163901 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:27.165195 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:27.165840 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:27.166235 [ 31773 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:27.166373 [ 31773 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:27.842578 [ 31757 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 6012ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:27.945374 [ 31755 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:27.948575 [ 31755 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:27.958047 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:27.959144 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:27.982366 [ 32448 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:27.982555 [ 32448 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:27.982871 [ 32448 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007144889 sec. (139.960 rows/sec., 3.28 KiB/sec.)
2026.05.06 08:49:27.983043 [ 32448 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:27.987401 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:27.988110 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:27.994923 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:27.995673 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:28.008183 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:28.011229 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:28.011783 [ 32285 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:28.048015 [ 31747 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.048424 [ 31747 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.048446 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:28.048770 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:28.048828 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.049039 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.051437 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.051487 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.053203 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.053287 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.053476 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:28.053719 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:28.053829 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:28.053887 [ 31754 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.054217 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:28.054279 [ 31754 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.054481 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.054558 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:28.054587 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:28.054714 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:28.054840 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:28.054871 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.055465 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:28.056182 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.056669 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.056954 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:28.057657 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.057664 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:28.057868 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:28.058068 [ 31745 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.058223 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:28.058440 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.058438 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:28.058741 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:28.058828 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.058881 [ 31740 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.059237 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:28.059482 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.060141 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:28.060277 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:28.060388 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:28.060823 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.061272 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.062561 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.063025 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.063422 [ 31739 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.063852 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:28.063904 [ 31739 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.064188 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.064210 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:28.064610 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.065474 [ 31743 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:28.065587 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:28.065741 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:28.066040 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.066568 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.067770 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:28.068265 [ 31743 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 990ms, ratio: 0.025641028, points per minute: 15.540017)
2026.05.06 08:49:28.068364 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.068595 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:28.068705 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:28.068807 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:28.070303 [ 31743 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:28.070356 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:28.070453 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:28.070553 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:28.071098 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:28.071184 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:28.071272 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:28.071443 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:28.071578 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:28.071776 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:28.073168 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:28.073845 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:28.074011 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:28.074113 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:28.074345 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:28.074886 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:28.076162 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:28.076306 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:28.076411 [ 31743 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:28.076613 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:28.076687 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:28.076808 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:28.076922 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:28.081627 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:28.081705 [ 31743 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:28.081800 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:28.081816 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:28.081908 [ 31743 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:28.082295 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:28.082874 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:28.083840 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:28.084237 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:28.085251 [ 31743 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:28.089077 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:28.089223 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:28.089373 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:28.089667 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.089895 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.090055 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:28.090155 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:28.090359 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:28.091308 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:28.091433 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:28.091589 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:28.091888 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.092089 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.092241 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:28.092332 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:28.092539 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:28.093625 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:28.093753 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:28.093894 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:28.094156 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.094351 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.094521 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:28.094614 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:28.094803 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:28.095378 [ 31731 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1336ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:28.095815 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:28.095946 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:28.096085 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:28.096331 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.096536 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.096688 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:28.096780 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:28.096967 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:28.098051 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:28.098176 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:28.098317 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:28.098581 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.098784 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.098939 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:28.099033 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:28.099226 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:28.100228 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:28.100354 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:28.100495 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:28.100764 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.100957 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.101106 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:28.101194 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:28.101397 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:28.102162 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:28.102278 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:28.102410 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:28.102669 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.102861 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:28.103005 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:28.103094 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:28.103281 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:28.103837 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:28.104474 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:28.104883 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:28.104974 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:28.108395 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:28.109865 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:28.110561 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:28.110997 [ 31732 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:28.111277 [ 31732 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.111654 [ 31732 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:28.111654 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:28.112018 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.113828 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.115551 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:28.116371 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:28.116483 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:28.116616 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:28.119393 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:28.121772 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:28.121929 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:28.125943 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:28.128328 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:28.128418 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:28.128559 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:28.128679 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:28.133663 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:28.133861 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:28.134396 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:28.135024 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:28.136019 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:28.136425 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:28.141240 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:28.141439 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:28.141615 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:28.141907 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.142140 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.142301 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:28.142400 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:28.142650 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:28.143735 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:28.143876 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:28.144021 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:28.144302 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.144525 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.144682 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:28.144777 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:28.144972 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:28.146072 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:28.146200 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:28.146356 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:28.146626 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.146823 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.146980 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:28.147075 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:28.147264 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:28.148315 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:28.148444 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:28.148603 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:28.148851 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.149049 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.149203 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:28.149296 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:28.149491 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:28.150715 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:28.150874 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:28.151022 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:28.151290 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.151519 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.151709 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:28.151810 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:28.152010 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:28.153014 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:28.153142 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:28.153283 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:28.153554 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.153751 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.153919 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:28.154029 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:28.154248 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:28.155049 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:28.155202 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:28.155366 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:28.155684 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.155889 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:28.156039 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:28.156131 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:28.156323 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:28.156930 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:28.157639 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:28.158075 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:28.158167 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:28.162665 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:28.164195 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:28.164945 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:28.165403 [ 31727 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:28.165645 [ 31727 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:28.839284 [ 31698 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.839860 [ 31698 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.839868 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:28.840241 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.840265 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:28.840688 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.843155 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.843218 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.845044 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.845086 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.845328 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:28.845522 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:28.845634 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:28.845637 [ 31691 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.846048 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:28.846137 [ 31691 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.846336 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:28.846355 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.846468 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:28.846602 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:28.846629 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:28.847032 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.847176 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:28.848036 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.848541 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:28.848799 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.849270 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:28.849464 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:28.849485 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.849828 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:28.849831 [ 31689 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.850296 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:28.850359 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:28.850579 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.851363 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.851908 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:28.852078 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:28.852196 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:28.852476 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.854191 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.854549 [ 31687 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.854869 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:28.854884 [ 31687 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.855127 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.855167 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:28.855484 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.856834 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.857157 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.857520 [ 31697 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:28.857628 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:28.857758 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:28.858417 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.858779 [ 31685 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.858995 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:28.859129 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:28.859395 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.860121 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:28.860235 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:28.860338 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:28.860521 [ 31697 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 775ms, ratio: 0.025641028, points per minute: 19.851118)
2026.05.06 08:49:28.860701 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:28.860792 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:28.861137 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.861386 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:28.861488 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:28.861607 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:28.862116 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:28.862828 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.863213 [ 31900 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.863477 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:28.863537 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:28.863590 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:28.863878 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.864203 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:28.864411 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:28.864829 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:28.865315 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:28.866156 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:28.866311 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:28.866577 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:28.866700 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:28.866806 [ 31697 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:28.866868 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.868637 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:28.870454 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:28.871951 [ 31697 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:28.872074 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:28.872190 [ 31697 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:28.872928 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:28.873008 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:28.873132 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:28.873247 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:28.874663 [ 31697 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:28.877027 [ 31697 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:28.878043 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:28.878232 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:28.878656 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:28.879180 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:28.880149 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:28.880564 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:28.885106 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:28.885254 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:28.885405 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:28.885706 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.885926 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.886087 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:28.886200 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:28.886411 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:28.887349 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:28.887479 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:28.887688 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:28.887953 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.888193 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.888361 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:28.888459 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:28.888670 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:28.889765 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:28.889893 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:28.890033 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:28.890283 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.890480 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.890667 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:28.890765 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:28.890955 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:28.891991 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:28.892120 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:28.892261 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:28.892529 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.892723 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.892876 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:28.892970 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:28.893158 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:28.894242 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:28.894386 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:28.894547 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:28.894804 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.895023 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.895179 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:28.895275 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:28.895471 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:28.896472 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:28.896614 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:28.896757 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:28.897009 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.897206 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.897357 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:28.897446 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:28.897652 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:28.898385 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:28.898519 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:28.898668 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:28.898916 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.899111 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:28.899274 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:28.899390 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:28.899714 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:28.900262 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:28.900953 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:28.901357 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:28.901448 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:28.905899 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:28.907224 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:28.907932 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:28.908336 [ 31865 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:28.908641 [ 31865 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:28.908965 [ 31865 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:28.909050 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:28.909433 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:28.912881 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:28.916755 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:28.917864 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:28.917987 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:28.918094 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:28.921081 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:28.923355 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:28.923521 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:28.927555 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:28.930560 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:28.930663 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:28.930789 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:28.930907 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:28.936087 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:28.936281 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:28.936892 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:28.937551 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:28.938790 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:28.939196 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:28.944578 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:28.944763 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:28.944920 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:28.945212 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.945429 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.945654 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:28.945783 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:28.945997 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:28.947029 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:28.947167 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:28.947350 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:28.947766 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.948066 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.948300 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:28.948443 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:28.948740 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:28.950340 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:28.950542 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:28.950743 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:28.951090 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.951310 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.951523 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:28.951690 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:28.951900 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:28.953089 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:28.953269 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:28.953476 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:28.953836 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.954083 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.954298 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:28.954428 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:28.954680 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:28.955959 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:28.956094 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:28.956241 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:28.956521 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.956775 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.956975 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:28.957100 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:28.957344 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:28.958359 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:28.958488 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:28.958654 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:28.958907 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.959105 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.959258 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:28.959348 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:28.959555 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:28.960368 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:28.960494 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:28.960665 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:28.960916 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.961115 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:28.961262 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:28.961354 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:28.961559 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:28.962088 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:28.962774 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:28.963214 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:28.963326 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:28.968043 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:28.969424 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:28.970093 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:28.970551 [ 31866 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:28.970776 [ 31866 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:28.992978 [ 32074 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:28.997396 [ 32142 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9027ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:28.999037 [ 32074 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:29.008170 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:29.009194 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:29.031052 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:29.031208 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:29.031532 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006580838 sec. (151.956 rows/sec., 3.56 KiB/sec.)
2026.05.06 08:49:29.031658 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:29.036037 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:29.036656 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:29.043170 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:29.043846 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:29.055637 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.058439 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.058898 [ 32282 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:29.238346 [ 32179 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 10035ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:29.326358 [ 32065 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.326866 [ 32065 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.327026 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:29.327277 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.327497 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.327756 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.330373 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.330446 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.332440 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.332525 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.332795 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:29.332968 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:29.333081 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:29.333082 [ 31684 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.333374 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:29.333520 [ 31684 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.333713 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.333856 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.333939 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:29.334066 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:29.334107 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.334180 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:29.334809 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:29.335477 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.335695 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.336155 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:29.336816 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:29.336901 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.337010 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:29.337103 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.337279 [ 32072 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.337354 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:29.337683 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:29.337722 [ 32072 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.337817 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:29.338082 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.338101 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.338443 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.338847 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:29.338979 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:29.339091 [ 32160 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:29.339562 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.339949 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.341128 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.341654 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.342149 [ 32063 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.342576 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:29.342625 [ 32063 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.342850 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.342915 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.343254 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.344280 [ 32160 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:29.344385 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:29.344513 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.344530 [ 32160 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:29.344913 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.345933 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:29.346492 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.346541 [ 32160 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 469ms, ratio: 0.025641028, points per minute: 32.803017)
2026.05.06 08:49:29.346700 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:29.346772 [ 32095 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:29.346811 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:29.346898 [ 32095 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:29.346913 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:29.347563 [ 32095 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:29.347734 [ 32095 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:29.347892 [ 32095 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:29.348569 [ 32095 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:29.349707 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:29.349940 [ 32095 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:29.350580 [ 32095 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:29.350768 [ 32095 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:29.351106 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:29.351478 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:29.352101 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:29.352251 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:29.352611 [ 32095 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:29.352743 [ 32095 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:29.352855 [ 32095 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:29.356305 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:29.358405 [ 32095 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:29.358538 [ 32095 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:29.358670 [ 32095 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:29.358685 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:29.358766 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:29.358889 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:29.359022 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:29.361118 [ 32095 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:29.363413 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:29.363655 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:29.364095 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:29.364517 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:29.365344 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:29.365732 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:29.370222 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:29.370367 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:29.370540 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:29.370825 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.371043 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.371204 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:29.371300 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:29.371523 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:29.372492 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:29.372645 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:29.372805 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:29.373067 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.373269 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.373426 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:29.373536 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:29.373729 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:29.374806 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:29.374960 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:29.375100 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:29.375350 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.375567 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.375756 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:29.375855 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:29.376047 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:29.377051 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:29.377178 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:29.377318 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:29.377584 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.377779 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.377933 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:29.378026 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:29.378213 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:29.379278 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:29.379406 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:29.379568 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:29.379870 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.380080 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.380236 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:29.380329 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:29.380553 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:29.381530 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:29.381654 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:29.381796 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:29.382047 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.382245 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.382394 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:29.382485 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:29.382703 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:29.383441 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:29.383581 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:29.383765 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:29.384028 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.384227 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:29.384387 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:29.384477 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:29.384687 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:29.385231 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:29.385918 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:29.386319 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:29.386410 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:29.390213 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:29.391496 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.392202 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.392625 [ 31894 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:29.392956 [ 31894 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.393265 [ 31894 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:29.393401 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.393781 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.395848 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.397980 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:29.398946 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:29.399081 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:29.399206 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:29.402340 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:29.405003 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:29.405166 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:29.409170 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:29.411850 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:29.411938 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:29.412085 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:29.412207 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:29.416901 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:29.417090 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:29.417708 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:29.418388 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:29.419453 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:29.419920 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:29.424459 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:29.424630 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:29.424783 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:29.425065 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.425281 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.425442 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:29.425555 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:29.425765 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:29.426778 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:29.426919 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:29.427092 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:29.427369 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.427644 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.427817 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:29.427919 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:29.428129 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:29.429294 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:29.429433 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:29.429606 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:29.429881 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.430093 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.430262 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:29.430367 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:29.430592 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:29.431740 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:29.431883 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:29.432034 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:29.432311 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.432538 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.432706 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:29.432807 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:29.433009 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:29.434170 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:29.434309 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:29.434478 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:29.434827 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.435061 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.435235 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:29.435340 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:29.435660 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:29.436749 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:29.436886 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:29.437041 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:29.437319 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.437549 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.437714 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:29.437811 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:29.438018 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:29.438841 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:29.438973 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:29.439119 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:29.439387 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.439655 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:29.439833 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:29.439932 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:29.440142 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:29.440731 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:29.441446 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:29.441908 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:29.442007 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:29.446041 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:29.447686 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.448444 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.448819 [ 31917 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:29.685539 [ 31883 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.686051 [ 31883 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.686064 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:29.686374 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.686491 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.686902 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.688572 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.689117 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.690322 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.690640 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:29.690828 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:29.690940 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:29.690964 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.690979 [ 32104 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.691368 [ 32104 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.691447 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:29.691581 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:29.691699 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.691723 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:29.691860 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:29.691907 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.692037 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.692454 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:29.694034 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.694156 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:29.694230 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.694998 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:29.695186 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:29.695642 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:29.696262 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:29.697275 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.697369 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.697737 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:29.697920 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:29.697996 [ 31927 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.698065 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:29.698362 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:29.698399 [ 31927 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.698694 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.698755 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.699034 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.700413 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.700924 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.702230 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.702645 [ 32064 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.702916 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.703020 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:29.703272 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.703314 [ 32152 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.703451 [ 32127 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:29.703581 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:29.703616 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.703780 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:29.703872 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.705327 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.705800 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.706385 [ 32127 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 345ms, ratio: 0.025641028, points per minute: 44.59309)
2026.05.06 08:49:29.706590 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:29.706690 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:29.707236 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:29.707296 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:29.707335 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:29.707440 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:29.707807 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.708001 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:29.708145 [ 32174 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.708184 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:29.708302 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:29.708410 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:29.708488 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.708800 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.709609 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:29.710417 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:29.710617 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:29.710875 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.710955 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:29.711183 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:29.711443 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:29.712840 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:29.712987 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:29.713103 [ 32127 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:29.713473 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:29.713553 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.713628 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:29.713947 [ 32156 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.714361 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.714800 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.717025 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.717716 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:29.719076 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:29.719267 [ 32127 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:29.719374 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:29.719523 [ 32127 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:29.720394 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:29.720483 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:29.720634 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:29.720757 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:29.722066 [ 32127 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:49:29.724247 [ 32127 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:29.725988 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:29.726223 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:29.726714 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:29.727311 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:29.728330 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:29.728780 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:29.734315 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:29.734494 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:29.734683 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:29.734999 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.735229 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.735398 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:29.735516 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:29.735787 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:29.736796 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:29.736934 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:29.737082 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:29.737381 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.737608 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.737772 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:29.737870 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:29.738075 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:29.739268 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:29.739406 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:29.739572 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:29.739870 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.740080 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.740238 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:29.740334 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:29.740546 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:29.741562 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:29.741702 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:29.741846 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:29.742137 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.742346 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.742523 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:29.742632 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:29.742827 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:29.743990 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:29.744131 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:29.744280 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:29.744561 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.744774 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.744935 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:29.745035 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:29.745238 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:29.746371 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:29.746526 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:29.746681 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:29.746946 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.747151 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.747308 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:29.747402 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:29.747662 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:29.748445 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:29.748585 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:29.748726 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:29.748985 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.749191 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:29.749346 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:29.749454 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:29.749670 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:29.750223 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:29.750948 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:29.751368 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:29.751462 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:29.755390 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:29.756851 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.757533 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.757982 [ 32092 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:29.758279 [ 32092 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:29.758633 [ 32092 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:29.758673 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:29.759001 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:29.760929 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:29.762846 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:29.763827 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:29.763953 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:29.764058 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:29.766857 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:29.769156 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:29.769310 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:29.773372 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:29.776025 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:29.776107 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:29.776230 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:29.776368 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:29.781465 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:29.781695 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:29.782188 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:29.782822 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:29.783966 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:29.784367 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:29.785287 [ 32157 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1690ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:29.789478 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:29.789673 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:29.789828 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:29.790117 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.790336 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.790554 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:29.790674 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:29.790886 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:29.791962 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:29.792102 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:29.792265 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:29.792581 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.792814 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.792976 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:29.793072 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:29.793272 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:29.794479 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:29.794652 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:29.794800 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:29.795061 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.795262 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.795434 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:29.795588 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:29.795828 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:29.796957 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:29.797094 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:29.797239 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:29.797527 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.797765 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.797928 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:29.798022 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:29.798215 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:29.799375 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:29.799572 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:29.799761 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:29.800033 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.800243 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.800410 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:29.800566 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:29.800792 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:29.801863 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:29.801996 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:29.802141 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:29.802411 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.802680 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.802844 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:29.802936 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:29.803132 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:29.804007 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:29.804139 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:29.804281 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:29.804614 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.804833 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:29.805001 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:29.805093 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:29.805289 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:29.805944 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:29.806715 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:29.807184 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:29.807286 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:29.812042 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:29.813445 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.814167 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.814738 [ 32166 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:29.814949 [ 32166 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:29.912491 [ 32176 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:29.919369 [ 32176 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:29.928968 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:29.930022 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:29.952065 [ 32329 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:29.952154 [ 32329 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:29.952321 [ 32329 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006330897 sec. (157.955 rows/sec., 3.70 KiB/sec.)
2026.05.06 08:49:29.952381 [ 32329 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:29.956150 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:29.956782 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:29.963524 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:29.964289 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:29.976057 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.978872 [ 32271 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:29.979325 [ 32271 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:30.541636 [ 32159 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:30.547854 [ 32159 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:30.557116 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:30.558166 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:30.584215 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=15, jobs=15). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:30.584599 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=15, jobs=15). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0917
8. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003afe2f18
9. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
10. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
11. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
12. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
13. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
14. ? @ 0x00007f6e2fbe4ac3
15. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:30.661401 [ 32148 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:30.661745 [ 32148 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:30.661927 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:30.661983 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:30.662336 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:30.662368 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:30.664985 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:30.665034 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:30.666911 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:30.666944 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:30.667214 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:30.667408 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:30.667551 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:30.667653 [ 32182 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:30.668167 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:30.668216 [ 32182 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:30.668324 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:30.668427 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:30.668490 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:30.668532 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:30.668560 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:30.668911 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:30.669255 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:30.670171 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:30.670648 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:30.670759 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:30.671477 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:30.671711 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:30.671784 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:30.672040 [ 32162 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:30.672102 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:30.672391 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:30.672433 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:30.672647 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:30.672773 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:30.672977 [ 32151 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:30.673457 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:30.673764 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:30.674209 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:30.674346 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:30.674464 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:30.674817 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:30.675401 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:30.676714 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:30.676958 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:30.677231 [ 32149 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:30.677707 [ 32149 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:30.677712 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:30.677982 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:30.678031 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:30.678254 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:30.679785 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:30.679856 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:30.679871 [ 32120 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:30.680028 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:30.680186 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:30.681403 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:30.681544 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:30.681968 [ 32080 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:30.682287 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:30.682397 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:30.682436 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:30.682497 [ 32120 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 958ms, ratio: 0.025641028, points per minute: 16.059097)
2026.05.06 08:49:30.682569 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:30.682687 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:30.682742 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:30.682779 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:30.683452 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:30.683559 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:30.683686 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:30.684181 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:30.684466 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:30.685392 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:30.685491 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:30.686136 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:30.686277 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:30.686451 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:30.686764 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:30.687157 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:30.687712 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:30.687878 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:30.688375 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:30.688497 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:30.688631 [ 32120 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:30.691922 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:30.693782 [ 32120 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:30.693875 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:30.693987 [ 32120 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:30.694216 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:30.694290 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:30.694407 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:30.694534 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:30.696395 [ 32120 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:30.698447 [ 32120 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:30.699334 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:30.699564 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:30.700056 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:30.700636 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:30.701580 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:30.701958 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:30.706521 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:30.706668 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:30.706844 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:30.707134 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.707346 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.707521 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:30.707660 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:30.707876 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:30.708807 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:30.708934 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:30.709084 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:30.709354 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.709567 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.709735 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:30.709827 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:30.710018 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:30.711090 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:30.711214 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:30.711352 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:30.711646 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.711852 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.712008 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:30.712103 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:30.712290 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:30.713274 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:30.713397 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:30.713550 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:30.713797 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.714007 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.714158 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:30.714250 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:30.714438 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:30.715515 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:30.715682 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:30.715825 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:30.716078 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.716280 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.716433 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:30.716544 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:30.716739 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:30.717729 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:30.717865 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:30.718002 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:30.718249 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.718444 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.718610 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:30.718700 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:30.718903 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:30.719690 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:30.719811 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:30.719945 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:30.720191 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.720387 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:30.720552 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:30.720640 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:30.720829 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:30.721370 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:30.722039 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:30.722439 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:30.722545 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:30.726661 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:30.728071 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:30.728742 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:30.729229 [ 32180 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:30.729635 [ 32180 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:30.729979 [ 32180 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:30.730120 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:30.730534 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:30.732386 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:30.734204 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:30.735053 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:30.735169 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:30.735274 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:30.738114 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:30.740455 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:30.740619 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:30.744618 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:30.747119 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:30.747204 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:30.747331 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:30.747451 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:30.749052 [ 32177 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9545ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:30.752474 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:30.752761 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:30.753209 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:30.753837 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:30.754809 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:30.755215 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:30.760172 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:30.760325 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:30.760479 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:30.760825 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.761065 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.761232 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:30.761333 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:30.761564 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:30.762596 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:30.762749 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:30.762913 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:30.763202 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.763428 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.763665 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:30.763776 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:30.763997 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:30.765254 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:30.765395 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:30.765574 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:30.765877 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.766089 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.766251 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:30.766351 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:30.766581 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:30.767724 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:30.767883 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:30.768029 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:30.768313 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.768543 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.768725 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:30.768848 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:30.769049 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:30.770217 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:30.770354 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:30.770534 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:30.770876 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.771096 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.771260 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:30.771359 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:30.771591 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:30.772743 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:30.772897 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:30.773046 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:30.773314 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.773542 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.773715 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:30.773817 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:30.774026 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:30.774935 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:30.775071 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:30.775216 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:30.775477 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.775826 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:30.775985 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:30.776079 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:30.776281 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:30.776917 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:30.777656 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:30.778103 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:30.778199 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:30.781763 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:30.783138 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:30.783855 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:30.784291 [ 31928 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:30.784479 [ 31928 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:30.913882 [ 32140 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1129ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:31.277303 [ 32163 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:31.284307 [ 32163 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:31.293801 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:31.294855 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:31.317169 [ 32427 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:31.317328 [ 32427 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:31.317660 [ 32427 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006922109 sec. (144.465 rows/sec., 3.39 KiB/sec.)
2026.05.06 08:49:31.317752 [ 32427 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:31.322009 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:31.322707 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:31.329641 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:31.330308 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:31.342769 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:31.346393 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:31.346988 [ 32285 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:31.773136 [ 32062 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:31.773598 [ 32062 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:31.773668 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:31.774432 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:31.774459 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:31.774759 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:31.777053 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:31.777110 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:31.778872 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:31.778893 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:31.779138 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:31.779294 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:31.779434 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:31.779486 [ 32038 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:31.779909 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:31.779903 [ 32055 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1031ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:31.780013 [ 32038 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:31.780040 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:31.780157 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:31.780287 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:31.780366 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:31.780377 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:31.780750 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:31.780914 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:31.782214 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:31.782311 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:31.782725 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:31.783128 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:31.783363 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:31.783740 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:31.784196 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:31.785193 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:31.785298 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:31.785644 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:31.785653 [ 32045 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:31.785807 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:31.785922 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:31.785998 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:31.786066 [ 32045 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:31.786338 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:31.786390 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:31.786768 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:31.788182 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:31.788584 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:31.789958 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:31.790379 [ 32043 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:31.790544 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:31.790706 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:31.790918 [ 32052 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:31.791205 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:31.791253 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:31.791282 [ 32194 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:31.791446 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:31.791663 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:31.791662 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:31.793537 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:31.793703 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:31.794493 [ 32194 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1096ms, ratio: 0.025641028, points per minute: 14.037059)
2026.05.06 08:49:31.794778 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:31.794890 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:31.795936 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:31.796070 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:31.796187 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:31.796412 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:31.796813 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:31.796860 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:31.797378 [ 32039 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:31.797621 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:31.797772 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:31.797823 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:31.797877 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:31.798095 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:31.798642 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:31.799495 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:31.799739 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:31.800163 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:31.800265 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:31.800665 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:31.800756 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:31.802377 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:31.802556 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:31.802692 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:31.803037 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:31.803136 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:31.803186 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:31.803568 [ 32031 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:31.804001 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:31.804322 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:31.806594 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:31.807253 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:31.809137 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:31.809795 [ 32194 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:31.809898 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:31.809985 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:31.810022 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:31.810076 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:31.810196 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:31.810310 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:31.812889 [ 32194 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 19ms, ratio: 0.025641028, points per minute: 809.7167)
2026.05.06 08:49:31.815293 [ 32194 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:31.815639 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:31.815836 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:31.816319 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:31.816932 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:31.817960 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:31.818340 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:31.822854 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:31.822999 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:31.823148 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:31.823426 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.823715 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.823903 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:31.824006 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:31.824215 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:31.825181 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:31.825308 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:31.825451 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:31.825724 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.825927 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.826079 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:31.826173 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:31.826364 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:31.827429 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:31.827574 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:31.827744 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:31.828063 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.828275 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.828448 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:31.828564 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:31.828758 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:31.829749 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:31.829874 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:31.830015 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:31.830265 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.830463 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.830631 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:31.830723 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:31.830909 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:31.832001 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:31.832131 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:31.832275 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:31.832540 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.832746 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.832915 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:31.833010 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:31.833205 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:31.834168 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:31.834289 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:31.834428 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:31.834694 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.834888 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.835035 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:31.835122 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:31.835309 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:31.836094 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:31.836215 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:31.836347 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:31.836609 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.836818 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:31.836963 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:31.837051 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:31.837239 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:31.837772 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:31.838428 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:31.838835 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:31.838926 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:31.842666 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:31.844021 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:31.844673 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:31.845069 [ 32057 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:31.845318 [ 32057 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:31.845647 [ 32057 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:31.845707 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:31.846029 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:31.848268 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:31.850494 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:31.851414 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:31.851537 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:31.851685 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:31.855193 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:31.858103 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:31.858350 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:31.863731 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:31.865817 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:31.865921 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:31.866076 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:31.866223 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:31.871416 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:31.871686 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:31.872289 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:31.872982 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:31.874049 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:31.874440 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:31.879146 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:31.879294 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:31.879444 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:31.879779 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.880006 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.880165 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:31.880262 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:31.880467 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:31.881425 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:31.881570 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:31.881712 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:31.881969 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.882170 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.882323 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:31.882418 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:31.882624 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:31.883732 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:31.883860 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:31.884001 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:31.884251 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.884445 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.884618 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:31.884726 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:31.884919 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:31.885907 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:31.886032 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:31.886170 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:31.886415 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.886621 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.886775 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:31.886868 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:31.887089 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:31.888365 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:31.888517 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:31.888667 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:31.888926 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.889134 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.889304 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:31.889401 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:31.889614 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:31.890609 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:31.890738 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:31.890884 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:31.891137 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.891337 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.891487 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:31.891591 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:31.891819 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:31.892638 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:31.892779 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:31.892934 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:31.893251 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.893531 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:31.893690 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:31.893786 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:31.893984 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:31.894562 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:31.895332 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:31.895986 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:31.896114 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:31.900643 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:31.902165 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:31.902880 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:31.903441 [ 31937 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:31.903776 [ 31937 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:32.181163 [ 32028 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:32.188112 [ 32028 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:32.197477 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:32.198552 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:32.221780 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:32.221923 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:32.222217 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.00729972 sec. (136.992 rows/sec., 3.21 KiB/sec.)
2026.05.06 08:49:32.222301 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:32.229452 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:32.230308 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:32.238643 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:32.239290 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:32.253285 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:32.256548 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:32.256873 [ 32275 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:32.592374 [ 32019 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:32.592835 [ 32019 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:32.592907 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:32.593275 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:32.593333 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:32.593715 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:32.596146 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:32.596206 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:32.598403 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:32.598496 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:32.598656 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:32.598787 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:32.598890 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:32.598896 [ 32024 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:32.599307 [ 32024 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:32.599305 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:32.599683 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:32.599735 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:32.599788 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:32.599788 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:32.599890 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:32.600098 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:32.600468 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:32.601907 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:32.601987 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:32.602082 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:32.602794 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:32.602990 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:32.603332 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:32.603853 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:32.604945 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:32.605019 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:32.605372 [ 32010 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:32.605495 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:32.605714 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:32.605774 [ 32010 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:32.605722 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:32.605890 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:32.606066 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:32.606108 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:32.606339 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:32.607789 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:32.607846 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:32.609467 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:32.609669 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:32.610021 [ 32008 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:32.610329 [ 32008 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:32.610340 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:32.610617 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:32.610674 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:32.610879 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:32.612569 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:32.612693 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:32.614378 [ 32047 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:32.614564 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:32.614667 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:32.614675 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:32.614778 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:32.615951 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:32.616132 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:32.616283 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:32.618647 [ 32047 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 803ms, ratio: 0.025641028, points per minute: 19.158924)
2026.05.06 08:49:32.618890 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:32.619029 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:32.619291 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:32.619744 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:32.619890 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:32.620035 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:32.620726 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:32.621593 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:32.621737 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:32.622320 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:32.623140 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:32.623390 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:32.623783 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:32.624221 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:32.625572 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:32.625765 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:32.625817 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:32.625919 [ 32047 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:32.628262 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:32.628341 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:32.628463 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:32.628594 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:32.633424 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:32.633642 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:32.634173 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:32.634365 [ 32047 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:32.634478 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:32.634663 [ 32047 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:32.634774 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:32.635939 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:32.636334 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:32.639041 [ 32047 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 21ms, ratio: 0.025641028, points per minute: 732.6008)
2026.05.06 08:49:32.641414 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:32.641578 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:32.641729 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:32.642011 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.642226 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.642386 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:32.642483 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:32.642707 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:32.643667 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:32.643799 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:32.643939 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:32.644195 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.644393 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.644578 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:32.644672 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:32.644916 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:32.646115 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:32.646247 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:32.646389 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:32.646659 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.646858 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.647015 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:32.647110 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:32.647300 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:32.648314 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:32.648441 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:32.648599 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:32.648847 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.649061 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.649215 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:32.649311 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:32.649500 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:32.650580 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:32.650706 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:32.650846 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:32.651100 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.651302 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.651457 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:32.651566 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:32.651792 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:32.652800 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:32.652941 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:32.653083 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:32.653337 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.653549 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.653700 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:32.653789 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:32.653979 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:32.654728 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:32.654853 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:32.655024 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:32.655318 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.655556 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:32.655811 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:32.655909 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:32.656102 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:32.656658 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:32.657297 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:32.657715 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:32.657806 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:32.661389 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:32.662717 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:32.663352 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:32.663766 [ 32007 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:32.664052 [ 32007 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:32.664353 [ 32007 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:32.664456 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:32.664923 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:32.666912 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:32.668883 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:32.669719 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:32.669839 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:32.669943 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:32.672366 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:32.672798 [ 32011 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1758ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:32.674215 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:32.674345 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:32.677592 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:32.680205 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:32.680292 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:32.680438 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:32.680614 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:32.685810 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:32.686055 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:32.686554 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:32.687137 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:32.688272 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:32.688726 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:32.693720 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:32.693871 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:32.694024 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:32.694306 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.694536 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.694697 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:32.694793 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:32.695000 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:32.695977 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:32.696109 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:32.696251 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:32.696521 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.696737 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.696892 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:32.696984 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:32.697174 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:32.698243 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:32.698369 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:32.698523 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:32.698773 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.698967 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.699122 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:32.699217 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:32.699406 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:32.700482 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:32.700631 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:32.700772 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:32.701039 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.701231 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.701382 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:32.701474 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:32.701678 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:32.702737 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:32.702863 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:32.703005 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:32.703256 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.703458 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.703677 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:32.703827 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:32.704038 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:32.705119 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:32.705259 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:32.705405 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:32.705690 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.705951 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.706120 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:32.706213 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:32.706408 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:32.707292 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:32.707421 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:32.707583 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:32.707914 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.708119 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:32.708268 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:32.708359 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:32.708571 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:32.709112 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:32.709783 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:32.710183 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:32.710275 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:32.713720 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:32.715218 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:32.716004 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:32.716428 [ 32005 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:32.716665 [ 32005 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:33.237243 [ 31999 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:33.243234 [ 31999 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:33.252427 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:33.253461 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:33.276746 [ 32444 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:33.276847 [ 32444 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:33.277025 [ 32444 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006492768 sec. (154.018 rows/sec., 3.61 KiB/sec.)
2026.05.06 08:49:33.277093 [ 32444 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:33.280985 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:33.281630 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:33.288387 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:33.289108 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:33.302661 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:33.306157 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:33.306742 [ 32285 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:33.696611 [ 31961 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:33.699792 [ 31961 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:33.708684 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:33.709715 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:33.731290 [ 32471 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:33.731429 [ 32471 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:33.731787 [ 32471 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006336718 sec. (157.810 rows/sec., 3.70 KiB/sec.)
2026.05.06 08:49:33.731878 [ 32471 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:33.735821 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:33.736473 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:33.743328 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:33.744064 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:33.756377 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:33.759399 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:33.759937 [ 32272 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:33.791095 [ 31954 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:33.791399 [ 31954 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:33.791482 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:33.791712 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:33.791907 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:33.792061 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:33.794366 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:33.794462 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:33.796199 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:33.796272 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:33.796456 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:33.796621 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:33.796722 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:33.796756 [ 31955 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:33.797173 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:33.797204 [ 31955 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:33.797469 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:33.797492 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:33.797551 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:33.797586 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:33.797710 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:33.797856 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:33.798275 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:33.799264 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:33.799499 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:33.799756 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:33.800536 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:33.800773 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:33.801069 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:33.801110 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:33.801160 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:33.801430 [ 31952 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:33.801548 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:33.801813 [ 31952 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:33.801901 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:33.802096 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:33.802294 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:33.802383 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:33.802950 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:33.803095 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:33.803208 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:33.804175 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:33.804172 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:33.805822 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:33.805943 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:33.806196 [ 31947 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:33.806660 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:33.806733 [ 31947 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:33.807007 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:33.807013 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:33.807281 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:33.808496 [ 31953 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:33.808614 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:33.808756 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:33.808819 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:33.808891 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:33.810612 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:33.810698 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:33.811148 [ 31953 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1172ms, ratio: 0.025641028, points per minute: 13.126806)
2026.05.06 08:49:33.811322 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:33.811413 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:33.811472 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:33.811626 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:33.811734 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:33.812081 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:33.812171 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:33.812262 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:33.812750 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:33.814153 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:33.814620 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:33.814886 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:33.815095 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:33.815468 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:33.815892 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:33.816960 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:33.817043 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:33.817112 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:33.817182 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:33.817292 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:33.821321 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:33.821432 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:33.821499 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:33.821642 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:33.821754 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:33.822728 [ 31953 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:33.822830 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:33.822951 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:33.825549 [ 31953 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:33.826124 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:33.826291 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:33.826713 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:33.827212 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:33.828119 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:33.828496 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:33.832359 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:33.832502 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:33.832665 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:33.832943 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.833137 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.833281 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:33.833369 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:33.833566 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:33.834453 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:33.834590 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:33.834718 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:33.834955 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.835133 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.835271 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:33.835370 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:33.835565 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:33.836588 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:33.836709 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:33.836836 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:33.837065 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.837241 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.837379 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:33.837465 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:33.837654 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:33.838536 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:33.838650 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:33.838776 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:33.839006 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.839181 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.839330 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:33.839415 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:33.839632 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:33.840598 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:33.840734 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:33.840873 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:33.841169 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.841406 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.841612 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:33.841717 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:33.841942 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:33.843025 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:33.843160 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:33.843310 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:33.843665 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.843953 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.844123 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:33.844221 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:33.844428 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:33.845241 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:33.845369 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:33.845534 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:33.845797 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.846004 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:33.846158 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:33.846255 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:33.846458 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:33.847116 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:33.847910 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:33.848377 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:33.848474 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:33.852126 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:33.853484 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:33.854167 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:33.854528 [ 31945 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:33.854802 [ 31945 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:33.854863 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:33.855178 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:33.856906 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:33.858479 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:33.859215 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:33.859315 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:33.859424 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:33.862256 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:33.864618 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:33.864776 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:33.868825 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:33.871247 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:33.871331 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:33.871457 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:33.871593 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:33.876564 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:33.876762 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:33.877267 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:33.877830 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:33.878868 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:33.879280 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:33.884184 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:33.884358 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:33.884533 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:33.884829 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.885053 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.885218 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:33.885318 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:33.885548 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:33.886527 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:33.886666 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:33.886814 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:33.887093 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.887304 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.887467 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:33.887583 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:33.887823 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:33.889028 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:33.889169 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:33.889316 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:33.889607 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.889817 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.889978 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:33.890076 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:33.890310 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:33.891420 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:33.891582 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:33.891780 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:33.892054 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.892295 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.892496 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:33.892631 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:33.892837 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:33.893958 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:33.894088 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:33.894233 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:33.894499 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.894733 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.894894 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:33.894992 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:33.895195 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:33.896226 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:33.896360 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:33.896523 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:33.896785 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.896987 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.897156 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:33.897249 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:33.897448 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:33.898220 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:33.898344 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:33.898525 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:33.898792 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.898990 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:33.899140 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:33.899233 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:33.899436 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:33.900209 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:33.900923 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:33.901342 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:33.901437 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:33.905990 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:33.907428 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:33.908233 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:33.908744 [ 31938 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:33.908930 [ 31938 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:34.039853 [ 32316 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:34.040370 [ 32316 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:34 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE736786D042C
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:34.040799 [ 32316 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:34.254912 [ 31926 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1583ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:34.402287 [ 32132 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:34.408544 [ 32132 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:34.417887 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:34.418977 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:34.440736 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=784, jobs=664). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:34.441250 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=784, jobs=664). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202eb394
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202d9869
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202db514
6. ./src/Common/ThreadPool.h:0: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000003b01a5ac
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000003b01bc2b
8. ./build_docker/./src/Interpreters/MutationsInterpreter.cpp:213: DB::isStorageTouchedByMutations(std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::MergeTreeData::IMutationsSnapshot const>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::vector<DB::MutationCommand, std::allocator<DB::MutationCommand>> const&, std::shared_ptr<DB::Context const>) @ 0x00000000350a3d7a
9. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:2185: DB::MutateTask::prepare() @ 0x0000000039fc4116
10. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:2054: DB::MutateTask::execute() @ 0x0000000039fc1b76
11. ./build_docker/./src/Storages/MergeTree/MutatePlainMergeTreeTask.cpp:0: DB::MutatePlainMergeTreeTask::executeStep() @ 0x000000003a5099dc
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
13. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
15. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
16. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
17. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
18. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
19. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
20. ? @ 0x00007f6e2fbe4ac3
21. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:34.568352 [ 31980 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 6726ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:34.826242 [ 31915 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:34.832609 [ 31915 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:34.842142 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:34.843245 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:34.855491 [ 31933 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:34.855985 [ 31933 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:34.856094 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:34.856339 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:34.856568 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:34.856691 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:34.859369 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:34.859450 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:34.861577 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:34.861662 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:34.861928 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:34.862130 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:34.862239 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:34.862338 [ 31909 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:34.862802 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:34.862844 [ 31909 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:34.863064 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:34.863171 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:34.863208 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:34.863206 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:34.863289 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:34.863647 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:34.863988 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:34.866191 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:34.866205 [ 32390 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:34.866336 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:34.866371 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:34.866385 [ 32390 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:34.866791 [ 32390 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006780488 sec. (147.482 rows/sec., 3.46 KiB/sec.)
2026.05.06 08:49:34.866884 [ 32390 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:34.867083 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:34.867332 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:34.867851 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:34.868390 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:34.869406 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:34.869493 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:34.869845 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:34.869991 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:34.870012 [ 31923 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:34.870105 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:34.870414 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:34.870572 [ 31923 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:34.870929 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:34.870940 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:34.871304 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:34.871892 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:34.872614 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:34.872915 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:34.873273 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:34.874686 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:34.875177 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:34.875581 [ 31903 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:34.875596 [ 32121 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:34.875816 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:34.875997 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:34.876031 [ 31903 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:34.875986 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:34.876325 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:34.876401 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:34.876644 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:34.878618 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:34.878655 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:34.878761 [ 32121 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1053ms, ratio: 0.025641028, points per minute: 14.610271)
2026.05.06 08:49:34.878953 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:34.879055 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:34.879711 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:34.879815 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:34.879918 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:34.880033 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:34.880427 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:34.880430 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:34.880754 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:34.880775 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:34.880943 [ 31918 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:34.881357 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:34.881695 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:34.881775 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:34.881908 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:34.882017 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:34.882078 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:34.882857 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:34.883057 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:34.883339 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:34.883632 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:34.883851 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:34.885087 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:34.885224 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:34.885225 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:34.885329 [ 32121 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:34.885747 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:34.888114 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:34.888279 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:34.890612 [ 32121 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:34.890720 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:34.890830 [ 32121 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:34.893086 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:34.893138 [ 32121 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:34.894422 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:34.895253 [ 32121 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:34.896693 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:34.897171 [ 32286 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:34.897313 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:34.897398 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:34.897560 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:34.897680 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:34.902671 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:34.902864 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:34.903336 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:34.903956 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:34.904965 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:34.905447 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:34.910093 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:34.910244 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:34.910400 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:34.910705 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.910948 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.911112 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:34.911213 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:34.911423 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:34.912428 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:34.912578 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:34.912737 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:34.912999 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.913205 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.913362 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:34.913455 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:34.913691 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:34.914855 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:34.914989 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:34.915136 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:34.915395 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.915660 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.915826 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:34.915923 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:34.916120 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:34.917135 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:34.917264 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:34.917406 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:34.917677 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.917878 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.918036 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:34.918131 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:34.918333 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:34.919418 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:34.919564 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:34.919751 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:34.920016 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.920235 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.920393 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:34.920489 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:34.920719 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:34.921712 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:34.921840 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:34.921986 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:34.922239 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.922440 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.922609 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:34.922700 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:34.922893 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:34.923696 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:34.923820 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:34.923958 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:34.924210 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.924410 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:34.924590 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:34.924683 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:34.924877 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:34.925428 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:34.926223 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:34.926665 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:34.926762 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:34.930716 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:34.932257 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:34.932980 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:34.933427 [ 32111 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:34.933834 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:34.933826 [ 32111 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:34.934190 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:34.936291 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:34.938069 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:34.938915 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:34.939030 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:34.939134 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:34.941902 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:34.944227 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:34.944380 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:34.948374 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:34.950558 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:34.950634 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:34.950755 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:34.950869 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:34.955906 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:34.956097 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:34.956611 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:34.957254 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:34.958379 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:34.958779 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:34.963330 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:34.963481 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:34.963687 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:34.963976 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.964194 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.964357 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:34.964457 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:34.964678 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:34.965627 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:34.965760 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:34.965903 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:34.966161 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.966379 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.966549 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:34.966642 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:34.966836 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:34.967952 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:34.968083 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:34.968224 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:34.968483 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.968698 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.968885 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:34.969023 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:34.969224 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:34.970246 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:34.970375 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:34.970577 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:34.970831 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.971028 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.971181 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:34.971274 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:34.971485 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:34.972621 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:34.972754 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:34.972898 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:34.973154 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.973362 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.973533 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:34.973629 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:34.973825 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:34.974844 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:34.974971 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:34.975113 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:34.975368 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.975583 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.975777 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:34.975867 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:34.976061 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:34.976861 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:34.976981 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:34.977116 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:34.977366 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.977591 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:34.977737 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:34.977828 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:34.978030 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:34.978564 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:34.979228 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:34.979727 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:34.979825 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:34.983776 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:34.985126 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:34.985822 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:34.986214 [ 31897 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:34.986403 [ 31897 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:35.062261 [ 31889 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 5824ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:35.096552 [ 31640 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #5
2026.05.06 08:49:35.096602 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:49:35.097020 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:35.133997 [ 32106 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:35.134406 [ 32106 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:35.134469 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:35.134769 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:35.134836 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:35.135054 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:35.137211 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:35.137276 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:35.139031 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:35.139080 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:35.139289 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:35.139530 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:35.139691 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:35.139749 [ 32100 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:35.140176 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:35.140230 [ 32100 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:35.140541 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:35.140542 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:35.140661 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:35.140763 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:35.140864 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:35.140876 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:35.141468 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:35.142460 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:35.142733 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:35.142903 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:35.143558 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:35.143796 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:35.144142 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:35.144274 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:35.144621 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:35.144803 [ 31872 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:35.145208 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:35.145580 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:35.145580 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:35.146050 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:35.146194 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:35.146308 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:35.147957 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:35.149898 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:35.150315 [ 31890 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:35.150734 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:35.150768 [ 31890 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:35.151078 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:35.151138 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:35.151519 [ 31871 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:35.151498 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:35.151659 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:35.151806 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:35.152919 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:35.153458 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:35.154334 [ 31871 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 259ms, ratio: 0.025641028, points per minute: 59.400063)
2026.05.06 08:49:35.154526 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:35.154619 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:35.154775 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:35.155149 [ 32103 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:35.155246 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:35.155336 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:35.155431 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:35.155428 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:35.155464 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:35.155823 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:35.155954 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:35.156358 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:35.156468 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:35.156595 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:35.157314 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:35.157841 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:35.158111 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:35.158311 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:35.158761 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:35.159320 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:35.159490 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:35.160611 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:35.160720 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:35.160762 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:35.160868 [ 31871 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:35.161135 [ 32089 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:35.161477 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:35.161772 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:35.162016 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:35.162166 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:35.163768 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:35.165753 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:35.166236 [ 32081 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:35.166658 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:35.166917 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:35.168164 [ 31871 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:35.168267 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:35.168388 [ 31871 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:35.169017 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:35.169721 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:35.170895 [ 31871 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:49:35.170958 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:35.171976 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:35.172024 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:35.172113 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:35.172118 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:35.172221 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:35.172285 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:35.172402 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:35.173176 [ 31871 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:35.174934 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:35.177408 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:35.177523 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:35.177616 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:35.177672 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:35.178137 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:35.178757 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:35.179904 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:35.180300 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:35.181964 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:35.184389 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:35.184465 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:35.184604 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:35.184718 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:35.185154 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:35.185314 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:35.185465 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:35.185768 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.185988 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.186149 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:35.186247 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:35.186452 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:35.187437 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:35.187581 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:35.187758 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:35.188034 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.188240 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.188394 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:35.188488 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:35.188709 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:35.189496 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:35.189699 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:35.189830 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:35.189961 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:35.190101 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:35.190207 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:35.190351 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.190564 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.190720 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:35.190771 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:35.190814 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:35.191005 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:35.191775 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:35.192018 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:35.192143 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:35.192237 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:35.192283 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:35.192572 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.192768 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.192920 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:35.193014 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:35.193201 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:35.194282 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:35.194410 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:35.194572 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:35.194830 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.195038 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.195195 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:35.195290 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:35.195489 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:35.196537 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:35.196689 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:35.196837 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:35.197096 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.197147 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:35.197297 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:35.197300 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.197463 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:35.197498 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:35.197611 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:35.197768 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.197817 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:35.197987 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.198150 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:35.198247 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:35.198456 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:35.198663 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:35.198793 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:35.198937 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:35.199195 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.199396 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:35.199413 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:35.199561 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:35.199565 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:35.199727 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:35.199766 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:35.199926 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:35.200033 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.200235 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.200392 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:35.200492 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:35.200501 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:35.200772 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:35.201295 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:35.202046 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:35.202045 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:35.202157 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:35.202227 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:35.202371 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:35.202653 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.202853 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.203012 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:35.203108 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:35.203309 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:35.204361 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:35.204536 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:35.204689 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:35.204939 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.205134 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.205286 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:35.205379 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:35.205582 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:35.206655 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:35.206782 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:35.206828 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:35.206923 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:35.207178 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.207384 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.207552 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:35.207692 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:35.207895 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:35.208348 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:35.208910 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:35.209037 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:35.209070 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:35.209193 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:35.209445 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.209612 [ 31861 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:35.209670 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.209862 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:35.209953 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:35.210143 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:35.210903 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:35.211024 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:35.211161 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:35.211408 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.211680 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:35.211832 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:35.211942 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:35.212136 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:35.212679 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:35.213364 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:35.213778 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:35.213870 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:35.217922 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:35.219233 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:35.219930 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:35.220208 [ 31881 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:35.269389 [ 31858 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:35.275344 [ 31858 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:35.284295 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:35.285307 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:35.306893 [ 32339 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:35.307064 [ 32339 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:35.307408 [ 32339 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006572878 sec. (152.140 rows/sec., 3.57 KiB/sec.)
2026.05.06 08:49:35.307501 [ 32339 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:35.312039 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:35.312746 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:35.319835 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:35.320478 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:35.332856 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:35.336304 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:35.336873 [ 32277 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:35.365443 [ 31857 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9573ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:35.479207 [ 31856 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:35.485202 [ 31856 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:35.494712 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:35.495809 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:35.522591 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=11, jobs=11). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:35.523254 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=11, jobs=11). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:35.671350 [ 31851 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 3891ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:35.708130 [ 31848 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:35.714772 [ 31848 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:35.724530 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:35.725635 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:35.752136 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:35.752738 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:36.110209 [ 31840 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1855ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:36.138309 [ 31836 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 10246ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:36.166185 [ 31837 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 7169ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:36.176764 [ 31833 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:36.177185 [ 31833 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:36.177172 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:36.177464 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:36.177457 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:36.177813 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:36.179387 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:36.179491 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:36.180950 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:36.181047 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:36.181173 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:36.181325 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:36.181431 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:36.181449 [ 31831 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:36.181776 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:36.181870 [ 31831 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:36.182106 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:36.182108 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:36.182266 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:36.182260 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:36.182395 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:36.182534 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:36.183136 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:36.183683 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:36.183821 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:36.184569 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:36.185325 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:36.185324 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:36.185551 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:36.185612 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:36.185691 [ 31849 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:36.185948 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:36.186056 [ 31849 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:36.186055 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:36.186334 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:36.186370 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:36.186434 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:36.186655 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:36.187739 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:36.187888 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:36.188004 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:36.188359 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:36.188434 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:36.190168 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:36.190244 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:36.190565 [ 31820 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:36.190904 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:36.190969 [ 31820 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:36.191000 [ 31826 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:36.191216 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:36.191277 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:36.191665 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:36.192917 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:36.193185 [ 31828 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:36.193283 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:36.193309 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:36.193430 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:36.194293 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:36.194879 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:36.194949 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:36.195059 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:36.195161 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:36.195558 [ 31828 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1022ms, ratio: 0.025641028, points per minute: 15.053441)
2026.05.06 08:49:36.195764 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:36.195860 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:36.196382 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:36.196472 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:36.196585 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:36.197074 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:36.197341 [ 31826 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:36.197915 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:36.198296 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:36.198890 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:36.199065 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:36.199335 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:36.199654 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:36.200231 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:36.200404 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:36.200748 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:36.200884 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:36.200992 [ 31828 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:36.204754 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:36.204792 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:36.204831 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:36.204892 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:36.204957 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:36.206298 [ 31828 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:36.206336 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:36.206396 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:36.206532 [ 31828 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:36.207352 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:36.208326 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:36.208434 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:36.208774 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:36.208950 [ 31828 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:36.209110 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:36.209980 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:36.210336 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:36.214799 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:36.214939 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:36.215084 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:36.215351 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.215590 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.215789 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:36.215891 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:36.216089 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:36.216991 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:36.217113 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:36.217250 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:36.217495 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.217704 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.217861 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:36.217951 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:36.218137 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:36.219254 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:36.219403 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:36.219558 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:36.219855 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.220049 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.220201 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:36.220291 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:36.220477 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:36.221416 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:36.221550 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:36.221682 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:36.221933 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.222122 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.222268 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:36.222357 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:36.222555 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:36.223595 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:36.223761 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:36.223909 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:36.224155 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.224353 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.224503 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:36.224610 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:36.224799 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:36.225739 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:36.225861 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:36.225996 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:36.226239 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.226428 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.226583 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:36.226670 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:36.226870 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:36.227597 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:36.227708 [ 32454 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:36.227763 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:36.227770 [ 32454 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:36.227907 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:36.227951 [ 32454 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.005277267 sec. (189.492 rows/sec., 4.44 KiB/sec.)
2026.05.06 08:49:36.228008 [ 32454 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:36.228146 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.228335 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:36.228475 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:36.228576 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:36.228763 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:36.229271 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:36.229977 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:36.230358 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:36.230444 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:36.230948 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:36.231440 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:36.233284 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:36.234629 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:36.235291 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:36.235813 [ 31822 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:36.236222 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:36.236278 [ 31822 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:36.236643 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:36.237102 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:36.237665 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:36.238234 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:36.239676 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:36.240369 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:36.240476 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:36.240592 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:36.243366 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:36.245752 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:36.245903 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:36.247675 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:36.250092 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:36.250503 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:36.251009 [ 32278 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:36.252361 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:36.252440 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:36.252599 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:36.252714 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:36.257313 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:36.257497 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:36.257969 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:36.258502 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:36.259467 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:36.259895 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:36.264479 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:36.264671 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:36.264822 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:36.265102 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.265330 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.265490 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:36.265607 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:36.265821 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:36.266742 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:36.266871 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:36.267012 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:36.267267 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.267467 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.267688 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:36.267784 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:36.267976 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:36.269035 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:36.269161 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:36.269300 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:36.269576 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.269782 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.269938 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:36.270032 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:36.270219 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:36.271209 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:36.271335 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:36.271475 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:36.271779 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.271981 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.272135 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:36.272228 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:36.272413 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:36.273497 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:36.273640 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:36.273793 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:36.274054 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.274260 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.274418 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:36.274530 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:36.274727 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:36.275767 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:36.275895 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:36.276039 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:36.276306 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.276503 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.276671 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:36.276760 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:36.276955 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:36.277741 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:36.277863 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:36.278001 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:36.278250 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.278446 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:36.278610 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:36.278715 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:36.278923 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:36.279447 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:36.280185 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:36.280604 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:36.280696 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:36.284003 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:36.285311 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:36.285984 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:36.286355 [ 31808 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:36.654688 [ 31817 ] {} <Trace> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.654748 [ 31806 ] {} <Trace> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.654826 [ 31799 ] {} <Trace> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.654884 [ 31804 ] {} <Trace> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.655530 [ 31795 ] {} <Trace> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.657294 [ 31791 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92) (MergerMutator): Nothing to merge in partition 0: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.657484 [ 31791 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92) (MergerMutator): Nothing to merge in partition 1: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.657596 [ 31791 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92) (MergerMutator): Nothing to merge in partition 2: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.657705 [ 31791 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92) (MergerMutator): Checked 3 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:36.657921 [ 31791 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.695739 [ 31785 ] {} <Trace> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.699412 [ 31810 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Nothing to merge in partition 20191001: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.699573 [ 31810 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Nothing to merge in partition 20191002: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.699725 [ 31810 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Nothing to merge in partition 20191003: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.699549 [ 31789 ] {} <Test> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.699857 [ 31810 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Checked 3 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:36.699968 [ 31789 ] {} <Trace> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:36.702038 [ 31784 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 0: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.702199 [ 31784 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 1: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.702291 [ 31784 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 2: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.702384 [ 31784 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 3: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.702470 [ 31784 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 4: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.702572 [ 31784 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 5: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.702657 [ 31784 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 6: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.702760 [ 31784 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 7: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.702846 [ 31784 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 8: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.702927 [ 31784 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 9: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.703031 [ 31784 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Checked 10 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:36.703208 [ 31784 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.706876 [ 31789 ] {} <Trace> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.707701 [ 31802 ] {} <Test> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3) (MergerMutator): Nothing to merge in partition 20191002: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.707787 [ 31810 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.707832 [ 31802 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:36.711139 [ 31802 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.715353 [ 31780 ] {} <Test> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.715591 [ 31780 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:36.720364 [ 31780 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.727321 [ 31779 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.736283 [ 31778 ] {} <Test> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:36.736485 [ 31778 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:36.740442 [ 31778 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:36.856262 [ 31772 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:36.859935 [ 31772 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:36.869560 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:36.870633 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:36.893367 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:36.893565 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:36.893881 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007120259 sec. (140.444 rows/sec., 3.29 KiB/sec.)
2026.05.06 08:49:36.893974 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:36.898452 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:36.899156 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:36.906754 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:36.907474 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:36.915005 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:36.917470 [ 32277 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:36.918024 [ 32277 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:37.390956 [ 31767 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:37.391365 [ 31767 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:37.391456 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:37.391853 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:37.391970 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:37.392295 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:37.394721 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:37.394786 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:37.396859 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:37.396899 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:37.397147 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:37.397330 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:37.397433 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:37.397570 [ 31761 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:37.397998 [ 31761 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:37.397997 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:37.398201 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:37.398302 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:37.398290 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:37.398424 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:37.398369 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:37.398724 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:37.399035 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:37.400383 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:37.400528 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:37.400571 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:37.401150 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:37.401353 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:37.401745 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:37.401987 [ 31755 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:37.402044 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:37.402248 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:37.402406 [ 31748 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:37.402759 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:37.403041 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:37.403295 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:37.403692 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:37.403822 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:37.403933 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:37.404734 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:37.406305 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:37.406736 [ 31752 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:37.407126 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:37.407149 [ 31752 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:37.407375 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:37.407429 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:37.407762 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:37.407806 [ 31755 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:37.409075 [ 31756 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:37.409166 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:37.409173 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:37.409329 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:37.409376 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:37.411065 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:37.411162 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:37.412031 [ 31756 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1203ms, ratio: 0.025641028, points per minute: 12.788542)
2026.05.06 08:49:37.412179 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:37.412208 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:37.412286 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:37.412296 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:37.412391 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:37.413011 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:37.413097 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:37.413185 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:37.413662 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:37.415205 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:37.415221 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:37.415918 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:37.416092 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:37.416415 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:37.416913 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:37.416939 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:37.417249 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:37.417342 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:37.418134 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:37.418135 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:37.418263 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:37.418364 [ 31756 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:37.419208 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:37.421829 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:37.421906 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:37.422024 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:37.422150 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:37.423465 [ 31756 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:37.423571 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:37.423721 [ 31756 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:37.426185 [ 31756 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:37.427496 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:37.427737 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:37.428137 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:37.428649 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:37.429549 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:37.429903 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:37.434697 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:37.434845 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:37.434995 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:37.435273 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.435487 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.435689 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:37.435805 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:37.436053 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:37.437061 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:37.437193 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:37.437339 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:37.437614 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.437817 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.437970 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:37.438063 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:37.438256 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:37.439352 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:37.439480 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:37.439678 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:37.439932 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.440133 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.440289 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:37.440397 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:37.440604 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:37.441038 [ 32395 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:37.441202 [ 32395 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:37.441542 [ 32395 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006993088 sec. (142.998 rows/sec., 3.35 KiB/sec.)
2026.05.06 08:49:37.441596 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:37.441636 [ 32395 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:37.441722 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:37.441863 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:37.442113 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.442340 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.442535 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:37.442652 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:37.442853 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:37.442972 [ 31745 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1333ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:37.444176 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:37.444318 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:37.444480 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:37.444788 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.445001 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.445159 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:37.445256 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:37.445454 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:37.446661 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:37.446826 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:37.447005 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:37.447315 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.447532 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:37.447543 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.447803 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:37.447895 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:37.448152 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:37.448278 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:37.449056 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:37.449221 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:37.449392 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:37.449700 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.449948 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:37.450138 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:37.450265 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:37.450527 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:37.451167 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:37.452132 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:37.452644 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:37.452779 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:37.455812 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:37.456447 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:37.456500 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:37.458166 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:37.459164 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:37.459756 [ 31740 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:37.460219 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:37.460268 [ 31740 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:37.460598 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:37.462311 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:37.463914 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:37.464563 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:37.464633 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:37.464697 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:37.467373 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:37.469698 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:37.469850 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:37.470151 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:37.473933 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:37.474326 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:37.474376 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:37.474457 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:37.474543 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:37.474534 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:37.475230 [ 32286 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:37.477987 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:37.478100 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:37.478578 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:37.479111 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:37.480182 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:37.480451 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:37.482968 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:37.483064 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:37.483161 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:37.483338 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.483472 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.483581 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:37.483656 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:37.483784 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:37.484357 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:37.484441 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:37.484540 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:37.484692 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.484810 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.484902 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:37.484957 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:37.485066 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:37.485736 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:37.485814 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:37.485898 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:37.486042 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.486158 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.486252 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:37.486317 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:37.486427 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:37.487029 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:37.487105 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:37.487189 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:37.487332 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.487445 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.487546 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:37.487622 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:37.487730 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:37.488351 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:37.488428 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:37.488525 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:37.488671 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.488789 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.488887 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:37.488944 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:37.489057 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:37.489649 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:37.489727 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:37.489811 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:37.489954 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.490069 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.490157 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:37.490209 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:37.490318 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:37.490767 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:37.490838 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:37.490920 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:37.491062 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.491181 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:37.491265 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:37.491318 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:37.491426 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:37.491774 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:37.492162 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:37.492397 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:37.492453 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:37.496157 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:37.496910 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:37.497276 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:37.497620 [ 31736 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:37.497854 [ 31736 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:37.616837 [ 31737 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1478ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:37.729461 [ 32318 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:37.729998 [ 32318 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:37 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE73754559473
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:37.730408 [ 32318 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:38.169353 [ 31732 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:38.169607 [ 31732 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:38.169818 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:38.169833 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:38.170319 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:38.170318 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:38.173148 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:38.173206 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:38.174859 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:38.174901 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:38.175141 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:38.175334 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:38.175442 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:38.175443 [ 31725 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:38.175917 [ 31725 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:38.175958 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:38.176196 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:38.176253 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:38.176329 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:38.176356 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:38.176458 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:38.176643 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:38.177122 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:38.178289 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:38.178343 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:38.178466 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:38.179263 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:38.179488 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:38.179870 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:38.179993 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:38.180022 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:38.180241 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:38.180414 [ 31719 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:38.180722 [ 31719 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:38.180822 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:38.180930 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:38.181091 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:38.181258 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:38.181656 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:38.181803 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:38.181918 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:38.182796 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:38.182861 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:38.184469 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:38.184577 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:38.184909 [ 31712 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:38.185264 [ 31712 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:38.185292 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:38.185477 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:38.185612 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:38.185672 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:38.187216 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:38.187351 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:38.187400 [ 31750 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:38.187504 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:38.187678 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:38.188975 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:38.189077 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:38.190207 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:38.190250 [ 31750 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 764ms, ratio: 0.025641028, points per minute: 20.136932)
2026.05.06 08:49:38.190374 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:38.190532 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:38.190577 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:38.190735 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:38.191460 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:38.191572 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:38.191700 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:38.192216 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:38.193596 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:38.193643 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:38.194340 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:38.194549 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:38.194840 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:38.195113 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:38.195903 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:38.196060 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:38.196324 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:38.196457 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:38.196579 [ 31750 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:38.200248 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:38.201996 [ 31750 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:38.202100 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:38.202221 [ 31750 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:38.202538 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:38.202618 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:38.202740 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:38.202875 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:38.204467 [ 31750 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:38.207738 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:38.207925 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:38.208325 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:38.208664 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:38.209602 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:38.210007 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:38.214601 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:38.214745 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:38.214894 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:38.215170 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.215383 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.215556 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:38.215702 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:38.215910 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:38.216844 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:38.216972 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:38.217126 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:38.217382 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.217612 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.217764 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:38.217856 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:38.218056 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:38.219118 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:38.219263 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:38.219399 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:38.219689 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.219887 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.220053 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:38.220145 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:38.220334 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:38.221018 [ 31713 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:38.221315 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:38.221473 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:38.221627 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:38.221875 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.222075 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.222225 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:38.222318 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:38.222504 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:38.223588 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:38.223761 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:38.223906 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:38.224171 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.224374 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.224545 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:38.224638 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:38.224750 [ 31713 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:38.224832 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:38.225803 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:38.225925 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:38.226075 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:38.226336 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.226544 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.226694 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:38.226783 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:38.226972 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:38.227746 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:38.227868 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:38.228011 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:38.228254 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.228449 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:38.228624 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:38.228712 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:38.228901 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:38.229405 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:38.230132 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:38.230538 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:38.230626 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:38.233589 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:38.234065 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:38.234698 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:38.235367 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:38.236320 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:38.236816 [ 31710 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:38.237419 [ 31710 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:38.237464 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:38.237811 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:38.242138 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:38.245124 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:38.246091 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:38.246216 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:38.246324 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:38.249180 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:38.251483 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:38.251694 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:38.255724 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:38.258226 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:38.258311 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:38.258477 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:38.258612 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:38.260369 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:38.260875 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0917
8. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003afe2f18
9. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
10. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
11. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
12. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
13. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
14. ? @ 0x00007f6e2fbe4ac3
15. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:38.263294 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:38.263481 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:38.264016 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:38.264556 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:38.265615 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:38.266025 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:38.270775 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:38.270934 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:38.271089 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:38.271370 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.271663 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.271833 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:38.271932 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:38.272138 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:38.273090 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:38.273219 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:38.273362 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:38.273633 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.273838 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.273993 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:38.274101 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:38.274294 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:38.275573 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:38.275798 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:38.275981 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:38.276272 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.276525 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.276729 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:38.276860 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:38.277095 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:38.278368 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:38.278564 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:38.278712 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:38.278996 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.279240 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.279450 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:38.279581 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:38.279859 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:38.281176 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:38.281314 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:38.281464 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:38.281751 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.281965 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.282126 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:38.282226 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:38.282426 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:38.283432 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:38.283641 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:38.283794 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:38.284056 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.284273 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.284430 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:38.284537 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:38.284734 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:38.285500 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:38.285640 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:38.285779 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:38.286028 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.286227 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:38.286375 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:38.286468 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:38.286677 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:38.287248 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:38.288187 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:38.288702 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:38.288803 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:38.293108 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:38.294493 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:38.295177 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:38.295546 [ 31702 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:38.695346 [ 31866 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:38.702560 [ 31866 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:38.711954 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:38.713036 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:38.738750 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=4, jobs=4). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:38.739247 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=4, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:38.903640 [ 32074 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:38.906648 [ 32074 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:38.915342 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:38.916368 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:38.925017 [ 32143 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1482ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:38.937549 [ 32375 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:38.937615 [ 32375 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:38.937761 [ 32375 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.005942998 sec. (168.265 rows/sec., 3.94 KiB/sec.)
2026.05.06 08:49:38.937804 [ 32375 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:38.941637 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:38.942229 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:38.948918 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:38.949610 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:38.960565 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:38.963457 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:38.963974 [ 32283 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:39.160855 [ 31899 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 4592ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:39.389267 [ 31684 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.389705 [ 31684 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.389802 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:39.390130 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:39.390240 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.390558 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.393316 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.393409 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.395731 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.395849 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.395997 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:39.396182 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:39.396313 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:39.396338 [ 32072 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.396783 [ 32072 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.396793 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:39.397125 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:39.397179 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:39.397195 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.397232 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:39.397458 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:39.397603 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.398026 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:39.399451 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.399574 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:39.399714 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.400347 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:39.400545 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:39.400952 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:39.401403 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:39.402264 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.402391 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.402669 [ 32063 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.402739 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:39.402882 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:39.402994 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:39.403010 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:39.403142 [ 32063 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.403442 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.403488 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:39.403817 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.405423 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.405521 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.407390 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.407501 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.407819 [ 32107 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.408137 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:39.408201 [ 32107 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.408257 [ 32061 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:39.408378 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:39.408453 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:39.408503 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:39.408552 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.408729 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.410554 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.410727 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.411414 [ 32061 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1207ms, ratio: 0.025641028, points per minute: 12.746161)
2026.05.06 08:49:39.411634 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:39.411729 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:39.412397 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:39.412485 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:39.412588 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:39.412593 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:39.412859 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.413159 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:39.413293 [ 31904 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.413468 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:39.413593 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:39.413636 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:39.413695 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:39.413917 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.414792 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:39.415534 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:39.415745 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:39.415809 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.416177 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:39.416716 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:39.416714 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:39.418128 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:39.418278 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:39.418386 [ 32061 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:39.418906 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.419014 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:39.419163 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:39.419261 [ 31886 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.419770 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:39.420171 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.422161 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.423174 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:39.424651 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.424894 [ 32061 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:39.424998 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:39.425124 [ 32061 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:39.425136 [ 31885 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.425453 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:39.425528 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:39.425549 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:39.425709 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:39.425819 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.425826 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:39.427795 [ 32061 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:49:39.427836 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.430095 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:39.430274 [ 32061 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:39.430883 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:39.430899 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:39.431037 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:39.431073 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:39.431139 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:39.431531 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:39.432231 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:39.433330 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:39.433716 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:39.434006 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:39.436281 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:39.436423 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:39.438383 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:39.438548 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:39.438700 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:39.438986 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.439198 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.439372 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:39.439470 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:39.439732 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:39.440350 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:39.440709 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:39.440858 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:39.441006 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:39.441294 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.441528 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.441692 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:39.441798 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:39.442000 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:39.443101 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:39.443231 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:39.443275 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:39.443373 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:39.443384 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:39.443553 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:39.443711 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.443735 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:39.443916 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.444075 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:39.444171 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:39.444363 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:39.445406 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:39.445557 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:39.445725 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:39.445991 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.446195 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.446353 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:39.446451 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:39.446680 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:39.448032 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:39.448214 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:39.448402 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:39.448709 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.448923 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.449082 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:39.449177 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:39.449312 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:39.449376 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:39.449492 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:39.450126 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:39.450368 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:39.450494 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:39.450662 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:39.450722 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:39.450920 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.451134 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.451287 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:39.451377 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:39.451587 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:39.451915 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:39.452355 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:39.452615 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:39.452745 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:39.452883 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:39.453185 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.453404 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:39.453575 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:39.453672 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:39.453868 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:39.454425 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:39.455133 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:39.455557 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:39.455707 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:39.457867 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:39.458046 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:39.458223 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:39.458567 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.458827 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.459036 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:39.459162 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:39.459307 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:39.459400 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:39.460644 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:39.460680 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:39.460824 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:39.461000 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:39.461318 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.461326 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:39.461585 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.461728 [ 32104 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:39.461781 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:39.461901 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:39.462136 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:39.463570 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:39.463789 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:39.463947 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:39.464215 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.464444 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.464643 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:39.464739 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:39.464930 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:39.465903 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:39.466030 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:39.466167 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:39.466419 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.466685 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.466873 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:39.466969 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:39.467159 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:39.468277 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:39.468407 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:39.468567 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:39.468826 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.469031 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.469185 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:39.469279 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:39.469472 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:39.470432 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:39.470577 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:39.470722 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:39.470975 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.471184 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.471332 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:39.471420 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:39.471707 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:39.472444 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:39.472579 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:39.472714 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:39.473012 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.473247 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:39.473447 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:39.473593 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:39.473833 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:39.474532 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:39.475440 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:39.476027 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:39.476157 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:39.480805 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:39.482264 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:39.482960 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:39.483437 [ 32133 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:39.483789 [ 32133 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:39.505291 [ 31867 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:39.511401 [ 31867 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:39.521409 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:39.522631 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:39.546481 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:39.546655 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:39.547045 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007046239 sec. (141.920 rows/sec., 3.33 KiB/sec.)
2026.05.06 08:49:39.547151 [ 32466 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:39.551683 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:39.552345 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:39.559255 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:39.560000 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:39.571915 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:39.574905 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:39.575403 [ 32276 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:39.947446 [ 31895 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.947977 [ 31895 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.948077 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:39.948349 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:39.948499 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.948766 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.951568 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.951707 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.953968 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.954134 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.954259 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:39.954387 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:39.954616 [ 32174 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.955053 [ 32174 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.955086 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:39.955329 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:39.955410 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:39.955427 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.955537 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:39.955677 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:39.955711 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.956262 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:39.957809 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.957893 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.957993 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:39.958881 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:39.959102 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:39.959159 [ 32155 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1034ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:39.959477 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:39.959852 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.959945 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.960070 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:39.960400 [ 32088 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.960808 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:39.960879 [ 32088 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.961172 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.961197 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:39.961473 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.961794 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:39.961947 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:39.962064 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:39.963070 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.963296 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.965318 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.965368 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.965992 [ 32105 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.966348 [ 32105 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:39.966355 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:39.966657 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.966708 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:39.966990 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:39.967371 [ 32171 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:39.967475 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:39.967676 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:39.969055 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.969200 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:39.970467 [ 32171 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 540ms, ratio: 0.025641028, points per minute: 28.49003)
2026.05.06 08:49:39.970687 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:39.970811 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:39.971458 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:39.971588 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:39.971674 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:39.971729 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:39.971868 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:39.972442 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:39.972446 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:39.972578 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:39.972682 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:39.974246 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:39.975062 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:39.975248 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:39.975664 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:39.975969 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:39.976162 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:39.977750 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:39.977880 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:39.977991 [ 32171 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:39.978673 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:39.978861 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:39.983757 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:39.985641 [ 32171 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:39.985756 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:39.985894 [ 32171 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:39.986142 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:39.986228 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:39.986361 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:39.986487 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:39.989338 [ 32171 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 19ms, ratio: 0.025641028, points per minute: 809.7167)
2026.05.06 08:49:39.991765 [ 32171 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:39.991917 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:39.992127 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:39.992606 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:39.993231 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:39.994315 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:39.994765 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:39.999325 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:39.999474 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:39.999675 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:39.999959 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.000199 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.000358 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:40.000455 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:40.000677 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:40.001612 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:40.001741 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:40.001894 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:40.002148 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.002350 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.002505 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:40.002611 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:40.002803 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:40.003916 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:40.004047 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:40.004201 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:40.004450 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.004659 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.004817 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:40.004913 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:40.005103 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:40.006085 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:40.006210 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:40.006348 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:40.006607 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.006801 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.006953 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:40.007046 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:40.007234 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:40.008339 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:40.008466 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:40.008626 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:40.008879 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.009083 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.009237 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:40.009330 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:40.009540 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:40.010480 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:40.010622 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:40.010764 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:40.011013 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.011208 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.011357 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:40.011445 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:40.011695 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:40.012438 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:40.012572 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:40.012705 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:40.012949 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.013143 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:40.013287 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:40.013375 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:40.013577 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:40.014086 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:40.014759 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:40.015152 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:40.015242 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:40.019441 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:40.020938 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:40.021633 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:40.022052 [ 32083 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:40.022339 [ 32083 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:40.022726 [ 32083 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:40.022781 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:40.023143 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:40.025133 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:40.026880 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:40.027767 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:40.027888 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:40.028000 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:40.030974 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:40.033463 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:40.033641 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:40.038043 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:40.040193 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:40.040271 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:40.040393 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:40.040525 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:40.045470 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:40.045677 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:40.046209 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:40.046876 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:40.048273 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:40.048695 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:40.053627 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:40.053784 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:40.053947 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:40.054249 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.054495 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.054691 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:40.054797 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:40.055019 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:40.056012 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:40.056144 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:40.056287 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:40.056566 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.056770 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.056924 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:40.057018 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:40.057210 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:40.058309 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:40.058437 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:40.058613 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:40.058868 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.059070 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.059229 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:40.059324 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:40.059529 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:40.060561 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:40.060693 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:40.060835 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:40.061091 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.061292 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.061451 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:40.061560 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:40.061753 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:40.062875 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:40.063009 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:40.063158 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:40.063420 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.063697 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.063865 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:40.063964 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:40.064168 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:40.065196 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:40.065329 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:40.065480 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:40.065765 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.065972 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.066131 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:40.066226 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:40.066440 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:40.067244 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:40.067368 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:40.067527 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:40.067813 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.068017 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:40.068169 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:40.068262 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:40.068455 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:40.069020 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:40.069731 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:40.070150 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:40.070245 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:40.074329 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:40.075705 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:40.076349 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:40.076826 [ 32183 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:40.077068 [ 32183 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:40.116697 [ 32146 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:40.122918 [ 32146 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:40.134367 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:40.135798 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:40.164632 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:40.165168 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0917
8. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003afe2f18
9. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
10. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
11. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
12. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
13. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
14. ? @ 0x00007f6e2fbe4ac3
15. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:40.235816 [ 32134 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 2619ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:40.482717 [ 32159 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:40.483139 [ 32159 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:40.483188 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:40.483471 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:40.483584 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:40.483908 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:40.485971 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:40.486059 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:40.487753 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:40.487880 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:40.488000 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:40.488135 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:40.488240 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:40.488382 [ 32158 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:40.488883 [ 32158 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:40.488904 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:40.489054 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:40.489171 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:40.489203 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:40.489226 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:40.489305 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:40.489479 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:40.489954 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:40.491265 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:40.491334 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:40.491650 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:40.492377 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:40.492634 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:40.492986 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:40.493106 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:40.493212 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:40.493562 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:40.493572 [ 32181 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:40.494030 [ 32181 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:40.494082 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:40.494378 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:40.494477 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:40.494740 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:40.495179 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:40.495368 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:40.495499 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:40.496414 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:40.496644 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:40.498490 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:40.498654 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:40.498971 [ 32164 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:40.499301 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:40.499377 [ 32164 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:40.499690 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:40.499716 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:40.499998 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:40.501465 [ 32098 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:40.501586 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:40.501690 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:40.501739 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:40.501740 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:40.503438 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:40.503542 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:40.504169 [ 32098 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 513ms, ratio: 0.025641028, points per minute: 29.989506)
2026.05.06 08:49:40.504295 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:40.504363 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:40.504403 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:40.504451 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:40.504504 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:40.505128 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:40.505213 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:40.505303 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:40.505808 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:40.507167 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:40.507267 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:40.507854 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:40.508038 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:40.508359 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:40.508905 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:40.509583 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:40.509737 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:40.510272 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:40.510397 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:40.510497 [ 32098 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:40.514124 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:40.515933 [ 32098 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:40.516032 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:40.516139 [ 32098 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:40.516444 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:40.516536 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:40.516656 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:40.516769 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:40.518640 [ 32098 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:40.521462 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:40.521681 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:40.522120 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:40.522737 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:40.523789 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:40.524151 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:40.528668 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:40.528815 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:40.528963 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:40.529242 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.529452 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.529629 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:40.529726 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:40.529930 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:40.530884 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:40.531012 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:40.531154 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:40.531412 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.531675 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.531833 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:40.531926 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:40.532120 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:40.533191 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:40.533318 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:40.533456 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:40.533726 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.533923 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.534080 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:40.534176 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:40.534366 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:40.535340 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:40.535465 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:40.535652 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:40.535921 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.536114 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.536267 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:40.536358 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:40.536563 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:40.537626 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:40.537755 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:40.537900 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:40.538154 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.538358 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.538530 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:40.538624 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:40.538818 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:40.539816 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:40.539959 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:40.540100 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:40.540354 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.540567 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.540717 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:40.540807 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:40.540999 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:40.541747 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:40.541867 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:40.542000 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:40.542248 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.542443 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:40.542605 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:40.542694 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:40.542885 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:40.543418 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:40.544113 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:40.544531 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:40.544622 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:40.549103 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:40.550448 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:40.551103 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:40.551685 [ 32162 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:40.552104 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:40.552135 [ 32162 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:40.552444 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:40.554558 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:40.556626 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:40.557525 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:40.557630 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:40.557728 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:40.560803 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:40.563307 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:40.563497 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:40.567571 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:40.569636 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:40.569713 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:40.569830 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:40.569939 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:40.574727 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:40.574912 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:40.575475 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:40.576155 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:40.577385 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:40.577801 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:40.582300 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:40.582443 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:40.582606 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:40.582875 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.583084 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.583238 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:40.583332 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:40.583549 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:40.584493 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:40.584643 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:40.584785 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:40.585041 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.585238 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.585405 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:40.585496 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:40.585703 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:40.586766 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:40.586893 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:40.587031 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:40.587279 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.587470 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.587681 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:40.587779 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:40.587966 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:40.588934 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:40.589059 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:40.589194 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:40.589440 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.589664 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.589813 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:40.589906 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:40.590090 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:40.591126 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:40.591229 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:40.591346 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:40.591569 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.591761 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.591891 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:40.591968 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:40.592127 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:40.592924 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:40.593029 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:40.593145 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:40.593366 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.593542 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.593669 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:40.593742 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:40.593899 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:40.594502 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:40.594614 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:40.594726 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:40.594928 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.595087 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:40.595205 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:40.595279 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:40.595434 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:40.595939 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:40.596481 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:40.596825 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:40.596898 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:40.601240 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:40.602294 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:40.602826 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:40.603234 [ 32169 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:40.603451 [ 32169 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:40.968881 [ 32120 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:40.974975 [ 32120 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:40.984444 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:40.985553 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:41.009591 [ 32443 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:41.009768 [ 32443 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:41.010128 [ 32443 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007152269 sec. (139.816 rows/sec., 3.28 KiB/sec.)
2026.05.06 08:49:41.010234 [ 32443 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:41.014923 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:41.015656 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:41.022622 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:41.023305 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:41.029187 [ 32180 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1070ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:41.035732 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:41.039252 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:41.039848 [ 32281 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:41.049971 [ 31928 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.050261 [ 31928 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.050356 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:41.050548 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.050763 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.050914 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.053045 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.053145 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.055027 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.055041 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.055306 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:41.055469 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:41.055590 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:41.055718 [ 32138 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.056094 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:41.056201 [ 32138 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.056223 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:41.056366 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:41.056492 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:41.056553 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.056572 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.056895 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.057070 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:41.058114 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.058379 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:41.058501 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.058981 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:41.059163 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:41.059463 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.059548 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:41.060131 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:41.061164 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.061551 [ 32191 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.061672 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:41.061833 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:41.061888 [ 32191 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.061912 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:41.061948 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:41.062095 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.062184 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.062236 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.063954 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.063982 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.065462 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.065622 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.065856 [ 31908 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.066281 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.066305 [ 31908 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.066569 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.066652 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:41.066988 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.067147 [ 32135 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:41.067247 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:41.067400 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:41.068825 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.068918 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.069897 [ 32135 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 552ms, ratio: 0.025641028, points per minute: 27.870682)
2026.05.06 08:49:41.070082 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:41.070180 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:41.070493 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.070613 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:41.070851 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:41.070929 [ 32077 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.070944 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:41.071089 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:41.071270 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.071421 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:41.071549 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.071549 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:41.071634 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:41.071716 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:41.073071 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:41.073558 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.073880 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:41.074075 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:41.074538 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:41.074741 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:41.074978 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:41.075959 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.076322 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:41.076525 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:41.076648 [ 32135 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:41.077046 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:41.077187 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:41.081468 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:41.083737 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:41.083764 [ 32135 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:41.083819 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:41.083901 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:41.083941 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:41.084037 [ 32135 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:41.084056 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:41.086961 [ 32135 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:49:41.089082 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:41.089266 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:41.089729 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:41.090016 [ 32135 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:41.090265 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:41.091246 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:41.091684 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:41.096173 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:41.096317 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:41.096465 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:41.096761 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.096972 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.097131 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:41.097227 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:41.097430 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:41.098356 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:41.098484 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:41.098639 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:41.098893 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.099092 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.099258 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:41.099364 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:41.099596 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:41.100687 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:41.100812 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:41.100951 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:41.101199 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.101392 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.101563 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:41.101657 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:41.101843 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:41.102799 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:41.102920 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:41.103055 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:41.103296 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.103500 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.103695 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:41.103790 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:41.103976 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:41.105029 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:41.105153 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:41.105292 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:41.105555 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.105758 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.105909 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:41.106002 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:41.106193 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:41.107139 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:41.107259 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:41.107399 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:41.107719 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.107915 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.108064 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:41.108153 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:41.108341 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:41.109084 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:41.109201 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:41.109331 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:41.109589 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.109781 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.109925 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:41.110012 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:41.110200 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:41.110733 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:41.111363 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:41.111810 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:41.111903 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:41.115716 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:41.117021 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:41.117666 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:41.118129 [ 32123 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:41.118471 [ 32123 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.118801 [ 32123 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:41.118860 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.119169 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.120972 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.122576 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:41.123330 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:41.123445 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:41.123571 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:41.126598 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:41.129114 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:41.129287 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:41.133804 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:41.136204 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:41.136285 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:41.136404 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:41.136533 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:41.141733 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:41.141941 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:41.142490 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:41.143249 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:41.144804 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:41.145044 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:41.149625 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:41.149776 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:41.149930 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:41.150214 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.150431 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.150608 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:41.150707 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:41.150912 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:41.152022 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:41.152194 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:41.152345 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:41.152633 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.152840 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.153012 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:41.153106 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:41.153331 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:41.154462 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:41.154614 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:41.154756 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:41.155014 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.155213 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.155371 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:41.155467 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:41.155714 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:41.156716 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:41.156845 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:41.156985 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:41.157236 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.157449 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.157620 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:41.157715 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:41.157905 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:41.159003 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:41.159133 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:41.159277 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:41.159553 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.159796 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.159957 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:41.160053 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:41.160250 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:41.161231 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:41.161360 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:41.161505 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:41.161849 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.162055 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.162210 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:41.162300 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:41.162523 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:41.163344 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:41.163470 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:41.163668 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:41.163927 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.164124 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:41.164273 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:41.164365 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:41.164574 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:41.165099 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:41.165815 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:41.166253 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:41.166347 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:41.170289 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:41.171644 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:41.172288 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:41.172747 [ 32062 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:41.173055 [ 32062 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:41.716644 [ 32038 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.717086 [ 32038 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.717181 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:41.717566 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.717632 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.718050 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.720044 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.720284 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.721878 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.722127 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.722174 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:41.722309 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:41.722415 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:41.722570 [ 32043 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.722938 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:41.722989 [ 32043 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.723163 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:41.723234 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.723256 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.723272 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:41.723482 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:41.723666 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.724218 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:41.725250 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.725474 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.725635 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:41.726309 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:41.726599 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:41.726945 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:41.726974 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.727230 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.727388 [ 32050 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.727448 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:41.727872 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:41.727933 [ 32050 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.728205 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.728234 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.728469 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.729208 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:41.729410 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:41.729547 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:41.730406 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.730451 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.732361 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.732408 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.732940 [ 32031 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.733362 [ 32031 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.733421 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:41.733671 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.733808 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.733966 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.735211 [ 32045 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:41.735343 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:41.735525 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:41.735766 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.735825 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.737702 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:41.737735 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.738120 [ 32044 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.738372 [ 32045 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 648ms, ratio: 0.025641028, points per minute: 23.741692)
2026.05.06 08:49:41.738506 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:41.738570 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.738577 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:41.738633 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:41.738734 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:41.738772 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:41.738911 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.739371 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:41.739475 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:41.739628 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:41.740197 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:41.740656 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.741559 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:41.741586 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:41.742287 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:41.742337 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:41.742482 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:41.742861 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:41.743302 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:41.743859 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:41.744004 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:41.744667 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:41.744844 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:41.744985 [ 32045 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:41.748012 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:41.751596 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:41.751770 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:41.751909 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:41.752010 [ 32045 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:41.752030 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:41.752162 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:41.752309 [ 32045 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:41.755140 [ 32045 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:49:41.757329 [ 32045 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:41.757702 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:41.757922 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:41.758283 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:41.758841 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:41.759802 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:41.760211 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:41.765468 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:41.765660 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:41.765814 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:41.766101 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.766321 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.766482 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:41.766596 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:41.766805 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:41.767996 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:41.768134 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:41.768281 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:41.768563 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.768821 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.768994 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:41.769090 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:41.769288 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:41.770535 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:41.770714 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:41.770870 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:41.771136 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.771335 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.771494 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:41.771706 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:41.771925 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:41.773024 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:41.773158 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:41.773303 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:41.773585 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.773833 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.774006 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:41.774104 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:41.774318 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:41.775470 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:41.775701 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:41.775852 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:41.776113 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.776334 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.776527 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:41.776626 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:41.776826 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:41.777817 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:41.777942 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:41.778086 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:41.778341 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.778557 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.778723 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:41.778814 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:41.779007 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:41.779796 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:41.779920 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:41.780057 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:41.780307 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.780502 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:41.780692 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:41.780782 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:41.780973 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:41.781498 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:41.782183 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:41.782604 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:41.782695 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:41.786502 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:41.787910 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:41.788555 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:41.788988 [ 32041 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:41.789320 [ 32041 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:41.789656 [ 32041 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:41.789766 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:41.790170 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:41.792604 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:41.794307 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:41.795119 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:41.795232 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:41.795338 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:41.798220 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:41.800539 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:41.800688 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:41.804658 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:41.807092 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:41.807174 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:41.807298 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:41.807416 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:41.812397 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:41.812609 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:41.813294 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:41.813904 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:41.815234 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:41.815690 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:41.820352 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:41.820524 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:41.820698 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:41.821029 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.821289 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.821485 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:41.821607 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:41.821823 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:41.822796 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:41.822928 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:41.823075 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:41.823337 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.823562 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.823766 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:41.823864 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:41.824060 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:41.824388 [ 32032 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 6762ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:41.825199 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:41.825331 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:41.825476 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:41.825751 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.825951 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.826111 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:41.826208 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:41.826402 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:41.827409 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:41.827554 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:41.827726 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:41.827982 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.828178 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.828331 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:41.828440 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:41.828650 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:41.829797 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:41.829927 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:41.830070 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:41.830423 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.830672 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.830834 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:41.830934 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:41.831135 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:41.832170 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:41.832301 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:41.832445 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:41.832718 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.832918 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.833087 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:41.833178 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:41.833370 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:41.834215 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:41.834344 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:41.834485 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:41.834755 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.834954 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:41.835102 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:41.835193 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:41.835387 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:41.835972 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:41.836648 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:41.837050 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:41.837141 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:41.841275 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:41.842604 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:41.843218 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:41.843681 [ 32027 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:41.997073 [ 32028 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:42.003400 [ 32028 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:42.012833 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:42.013897 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:42.035972 [ 32421 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:42.036091 [ 32421 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:42.036311 [ 32421 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006796189 sec. (147.141 rows/sec., 3.45 KiB/sec.)
2026.05.06 08:49:42.036376 [ 32421 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:42.040501 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:42.041134 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:42.047861 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:42.048542 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:42.060390 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:42.063248 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:42.063763 [ 32285 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:42.070284 [ 32025 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.070714 [ 32025 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.070765 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:42.071020 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.071147 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.071373 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.073863 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.073931 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.075756 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.075866 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.075972 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:42.076081 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:42.076166 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:42.076396 [ 32024 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.076819 [ 32024 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.076819 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:42.076859 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:42.076943 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:42.077042 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:42.077061 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.077128 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.077352 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.077498 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:42.079013 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:42.079103 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.079327 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.079805 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:42.080048 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:42.080385 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:42.080935 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:42.081928 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.082001 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.082311 [ 32010 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.082324 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:42.082499 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:42.082622 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:42.082677 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:42.082741 [ 32010 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.083020 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.083042 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.083312 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.084994 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.085058 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.086838 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.086899 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.087390 [ 32012 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.087737 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:42.087790 [ 32012 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.087799 [ 32023 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:42.087912 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:42.088044 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.088065 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:42.088103 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.088296 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.089822 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.089950 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.090456 [ 32023 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 333ms, ratio: 0.025641028, points per minute: 46.200047)
2026.05.06 08:49:42.090646 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:42.090732 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:42.091353 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:42.091439 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:42.091560 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:42.091671 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:42.091722 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.092059 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:42.092112 [ 32016 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.092438 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.092547 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:42.092657 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:42.092747 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.092762 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:42.093430 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:42.094450 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:42.094649 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:42.094954 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:42.095349 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:42.095800 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:42.096291 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.096698 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:42.096838 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:42.096940 [ 32023 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:42.098075 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.098257 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:42.098407 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:42.098464 [ 32004 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.098857 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.099139 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.100925 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.102614 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.102991 [ 32013 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.103335 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.103424 [ 32023 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:42.103547 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:42.103684 [ 32023 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:42.103767 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.105018 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:42.105717 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.106014 [ 32023 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:49:42.107256 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:42.107332 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:42.107454 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:42.107533 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:42.107589 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:42.108054 [ 32023 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:42.108437 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:42.108568 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:42.108671 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:42.111348 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:42.112255 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:42.112442 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:42.112938 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:42.113444 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:42.113644 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:42.113783 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:42.114368 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:42.114765 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:42.117668 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:42.119362 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:42.119542 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:42.119724 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:42.120016 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.120206 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:42.120234 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.120305 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:42.120422 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:42.120431 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:42.120545 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:42.120610 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:42.120755 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:42.121712 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:42.121844 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:42.121988 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:42.122247 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.122449 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.122623 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:42.122717 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:42.122910 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:42.124092 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:42.124229 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:42.124373 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:42.124651 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.124852 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.125012 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:42.125108 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:42.125300 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:42.125439 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:42.125639 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:42.126223 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:42.126303 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:42.126432 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:42.126589 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:42.126722 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:42.126843 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.127044 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.127229 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:42.127331 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:42.127565 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:42.127824 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:42.128220 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:42.128782 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:42.128923 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:42.129078 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:42.129358 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.129594 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.129763 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:42.129866 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:42.130079 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:42.131142 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:42.131275 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:42.131429 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:42.131772 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.131992 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.132168 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:42.132266 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:42.132472 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:42.132860 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:42.133010 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:42.133162 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:42.133309 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:42.133438 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:42.133446 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.133604 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:42.133681 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.133843 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:42.133875 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.133941 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:42.134119 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:42.134148 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:42.134278 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:42.134376 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:42.134597 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:42.135083 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:42.135173 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:42.135211 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:42.135354 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:42.135699 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.135931 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.136006 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:42.136091 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:42.136188 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:42.136382 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:42.136468 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:42.136589 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:42.137496 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:42.137640 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:42.137782 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:42.138031 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.138226 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.138380 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:42.138474 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:42.138679 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:42.139680 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:42.139809 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:42.139947 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:42.140101 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:42.140197 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.140407 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.140577 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:42.140670 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:42.140857 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:42.141769 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:42.141931 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:42.142058 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:42.142198 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:42.142450 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.142542 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:42.142671 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.142827 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:42.142923 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:42.143118 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:42.144117 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:42.144245 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:42.144386 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:42.144652 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.144848 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.145015 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:42.145104 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:42.145294 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:42.146067 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:42.146185 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:42.146320 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:42.146579 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.146772 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.146919 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:42.147008 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:42.147198 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:42.147789 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:42.148468 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:42.148892 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:42.148984 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:42.156232 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:42.157443 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:42.158102 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:42.158488 [ 32017 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:42.158727 [ 32017 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:42.264853 [ 32001 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1235ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:42.807374 [ 32029 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.807915 [ 32029 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.807953 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:42.808156 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.808255 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.808308 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.810994 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.811031 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.812979 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.813016 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.813185 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:42.813329 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:42.813434 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:42.813566 [ 31993 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.814099 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:42.814144 [ 31993 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.814239 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:42.814365 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:42.814395 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.814467 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.814495 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:42.814538 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.815158 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:42.816689 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.816899 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.817081 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:42.817986 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:42.818192 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:42.818557 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:42.818779 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.818948 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.819145 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:42.819160 [ 32003 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.819636 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.819773 [ 32003 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.820132 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.820257 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:42.820690 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.820894 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:42.821036 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:42.821153 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:42.822254 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.822820 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.824591 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.825027 [ 31990 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.825235 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.825360 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.825556 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.827287 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.827280 [ 31996 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:42.827423 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:42.827585 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:42.829236 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.829598 [ 31981 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.830016 [ 31981 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.830043 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:42.830326 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.830393 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.830506 [ 31996 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 722ms, ratio: 0.025641028, points per minute: 21.308332)
2026.05.06 08:49:42.830713 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:42.830739 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.830815 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:42.831573 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:42.831733 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:42.831840 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:42.832310 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.832407 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:42.832570 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.834105 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:42.834253 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:42.834367 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:42.834936 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:42.835162 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:42.835198 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:42.835314 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:42.835422 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:42.835549 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:42.836050 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:42.837401 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:42.837615 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:42.837743 [ 31996 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:42.838311 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:42.840754 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:42.840911 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:42.844316 [ 31996 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:42.844439 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:42.844628 [ 31996 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:42.847425 [ 31996 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:49:42.847808 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:42.850830 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:42.850917 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:42.851050 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:42.851169 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:42.851971 [ 31996 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 5ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:42.857008 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:42.857199 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:42.857699 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:42.858278 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:42.859287 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:42.859730 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:42.864793 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:42.864949 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:42.865100 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:42.865386 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.865662 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.865831 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:42.865929 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:42.866137 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:42.867139 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:42.867270 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:42.867413 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:42.867721 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.867930 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.868085 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:42.868178 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:42.868368 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:42.869505 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:42.869690 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:42.869833 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:42.870090 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.870291 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.870452 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:42.870572 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:42.870780 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:42.871822 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:42.871953 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:42.872093 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:42.872345 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.872564 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.872741 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:42.872846 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:42.873044 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:42.874170 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:42.874302 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:42.874446 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:42.874739 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.874956 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.875113 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:42.875211 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:42.875410 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:42.876441 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:42.876585 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:42.876729 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:42.876983 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.877200 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.877371 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:42.877464 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:42.877699 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:42.878473 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:42.878616 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:42.878779 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:42.879104 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.879365 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:49:42.879585 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:42.879737 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:42.879986 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:42.880653 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:42.881351 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:42.881830 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:42.881930 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:42.885712 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:42.886301 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:42.886606 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:42.887073 [ 31976 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:42.887424 [ 31976 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:42.887885 [ 31976 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:42.887926 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:42.888286 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:42.890086 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:42.892412 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:42.893311 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:42.893423 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:42.893541 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:42.896345 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:42.898643 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:42.898792 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:42.902777 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:42.905264 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:42.905352 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:42.905478 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:42.905616 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:42.910623 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:42.910818 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:42.911330 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:42.911968 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:42.913098 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:42.913558 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:42.918595 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:42.918769 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:42.918929 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:42.919229 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.919502 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.919792 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:42.919903 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:42.920119 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:42.921191 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:42.921349 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:42.921501 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:42.921796 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.922006 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.922166 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:42.922262 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:42.922460 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:42.923687 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:42.923828 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:42.923987 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:42.924249 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.924450 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.924631 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:42.924729 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:42.924921 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:42.926076 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:42.926213 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:42.926357 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:42.926636 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.926835 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.926992 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:42.927087 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:42.927278 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:42.928446 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:42.928601 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:42.928749 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:42.929018 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.929259 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.929464 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:42.929590 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:42.929799 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:42.930892 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:42.931039 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:42.931191 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:42.931489 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.931761 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.931925 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:42.932021 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:42.932239 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:42.933140 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:42.933280 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:42.933427 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:42.933713 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.933920 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:42.934073 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:42.934167 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:42.934368 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:42.934958 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:42.935762 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:42.936198 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:42.936295 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:42.939777 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:42.940861 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:42.941457 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:42.941936 [ 31977 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:42.942159 [ 31977 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:42.964152 [ 31974 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:42.970275 [ 31974 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:42.980527 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:42.981693 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:43.005465 [ 32319 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:43.005659 [ 32319 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:43.005996 [ 32319 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006870608 sec. (145.548 rows/sec., 3.41 KiB/sec.)
2026.05.06 08:49:43.006099 [ 32319 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:43.010588 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:43.011242 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:43.018473 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:43.019200 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:43.031377 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.034383 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.034883 [ 32278 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:43.329841 [ 31985 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1066ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:43.400732 [ 31966 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.401159 [ 31966 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.401304 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:43.401576 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.401788 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.402059 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.404148 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.404330 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.406186 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.406198 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.406532 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:43.406736 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:43.406854 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:43.406886 [ 31961 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.407289 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:43.407334 [ 31961 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.407676 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.407708 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.407738 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:43.407852 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:43.407951 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.407975 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:43.408665 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:43.409676 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.409756 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.410098 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:43.410791 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:43.410979 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:43.411345 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:43.411469 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.411468 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.411906 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:43.411917 [ 31957 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.412464 [ 31957 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.412483 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.412750 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:43.412855 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.413042 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.413296 [ 31954 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:43.413569 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:43.413720 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:43.413848 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:43.414830 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.414926 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.416908 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.416948 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.417485 [ 31949 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.417832 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.417893 [ 31949 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.418158 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.418211 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:43.418567 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.419046 [ 31964 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:43.419146 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:43.419293 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:43.419649 [ 31954 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:43.420171 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.420705 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.421874 [ 31964 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 570ms, ratio: 0.025641028, points per minute: 26.990555)
2026.05.06 08:49:43.421971 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.422359 [ 31947 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.422697 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:43.422736 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.423056 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.423649 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:43.423764 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:43.423839 [ 31964 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:43.423867 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:43.423996 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:43.424096 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:43.424714 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:43.424811 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:43.424892 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.424911 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:43.425453 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:43.426795 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:43.426973 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.426996 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:43.427845 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:43.428030 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:43.428376 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:43.428834 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:43.429169 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:43.429317 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:43.430241 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:43.430366 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:43.430474 [ 31964 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:43.431400 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:43.432812 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:43.433433 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:43.435842 [ 31964 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:43.435938 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:43.435943 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:43.436037 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:43.436133 [ 31964 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:43.436173 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:43.436292 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:43.438846 [ 31964 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:43.440992 [ 31964 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:43.441582 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:43.441769 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:43.442186 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:43.442746 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:43.443660 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:43.444108 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:43.448674 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:43.448822 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:43.448974 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:43.449259 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.449474 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.449652 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:43.449751 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:43.449959 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:43.451079 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:43.451240 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:43.451409 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:43.451750 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.451977 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.452139 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:43.452238 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:43.452466 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:43.454010 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:43.454205 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:43.454377 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:43.454667 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.454870 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.455031 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:43.455128 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:43.455320 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:43.456422 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:43.456571 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:43.456711 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:43.456963 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.457158 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.457323 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:43.457414 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:43.457620 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:43.458694 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:43.458822 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:43.458966 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:43.459220 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.459216 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:43.459418 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:43.459429 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.459676 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:43.459775 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:43.459825 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006687519 sec. (149.532 rows/sec., 3.50 KiB/sec.)
2026.05.06 08:49:43.459925 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:43.459973 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:43.460941 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:43.461065 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:43.461223 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:43.461473 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.461686 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.461838 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:43.461926 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:43.462116 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:43.462866 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:43.462986 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:43.463121 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:43.463369 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.463579 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:43.463761 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:43.463851 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:43.464043 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:43.464598 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:43.464655 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:43.465300 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:43.465624 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:43.465716 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:43.465808 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:43.469148 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:43.470563 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.471210 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.471716 [ 31922 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.472127 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.472183 [ 31922 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:43.472521 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.473330 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:43.474085 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.474109 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:43.475481 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:43.476207 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:43.476317 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:43.476420 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:43.479181 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:43.481582 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:43.481736 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:43.485801 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:43.485807 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.488747 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:43.488850 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:43.488975 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:43.489093 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:43.489330 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.489858 [ 32278 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:43.493878 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:43.494068 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:43.494526 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:43.495099 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:43.496191 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:43.496607 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:43.501148 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:43.501320 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:43.501488 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:43.501841 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.502074 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.502239 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:43.502337 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:43.502562 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:43.503576 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:43.503779 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:43.503938 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:43.504226 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.504434 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.504609 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:43.504703 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:43.504900 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:43.506018 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:43.506149 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:43.506291 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:43.506560 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.506759 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.506915 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:43.507009 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:43.507201 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:43.508237 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:43.508369 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:43.508525 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:43.508779 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.508974 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.509129 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:43.509235 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:43.509424 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:43.510505 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:43.510648 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:43.510793 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:43.511051 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.511258 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.511413 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:43.511525 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:43.511755 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:43.512739 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:43.512866 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:43.513009 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:43.513266 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.513468 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.513655 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:43.513751 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:43.513953 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:43.514712 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:43.514834 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:43.514970 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:43.515218 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.515414 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.515574 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:43.515706 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:43.515908 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:43.516440 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:43.517254 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:43.517692 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:43.517784 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:43.521426 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:43.522755 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.523396 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.523907 [ 31920 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:43.524160 [ 31920 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:43.545647 [ 31980 ] {} <Test> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:43.545921 [ 31980 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:43.546095 [ 31980 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:43.546131 [ 31919 ] {} <Test> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:43.546298 [ 31919 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:43.546457 [ 31919 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:43.557458 [ 31916 ] {} <Test> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:43.557715 [ 31916 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:43.557934 [ 31916 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:43.561192 [ 31933 ] {} <Trace> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:43.572048 [ 31905 ] {} <Test> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:43.572301 [ 31905 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:43.573868 [ 32119 ] {} <Test> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:43.574106 [ 32119 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:43.574310 [ 32119 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:43.576165 [ 31905 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:43.595721 [ 31902 ] {} <Test> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:49:43.595928 [ 31902 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:49:43.596071 [ 31902 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:49:43.628048 [ 31901 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:43.634126 [ 31901 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:43.643392 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:43.644479 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:43.667717 [ 32353 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:43.667878 [ 32353 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:43.668194 [ 32353 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006599038 sec. (151.537 rows/sec., 3.55 KiB/sec.)
2026.05.06 08:49:43.668285 [ 32353 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:43.669656 [ 31896 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.669908 [ 31896 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.669979 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:43.670130 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.670316 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.670473 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.672255 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.672327 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.672915 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:43.673598 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:43.673728 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.673831 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.673940 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:43.674096 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:43.674202 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:43.674251 [ 32113 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.674789 [ 32113 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.674818 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:43.674829 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:43.674925 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:43.675043 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:43.675110 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.675210 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.675439 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.675689 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:43.676805 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.676897 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:43.676956 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.677482 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:43.677715 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:43.678002 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:43.678223 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.678375 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:43.679446 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.679586 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:43.679753 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:43.679897 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:43.679900 [ 32111 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.680290 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:43.680342 [ 32111 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.680614 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.680620 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.680768 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:43.680963 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.681487 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:43.682069 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.682417 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.683466 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.683898 [ 31897 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.683912 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.684366 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:43.684438 [ 31897 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.684741 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.684795 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.685140 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.685703 [ 32115 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:43.685829 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:43.686014 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:43.687119 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.687202 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.691824 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:43.691892 [ 32115 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 251ms, ratio: 0.025641028, points per minute: 61.29329)
2026.05.06 08:49:43.692017 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:43.692083 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:43.694833 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.695240 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.695359 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:43.695439 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:43.695491 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:43.695572 [ 31878 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.695542 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:43.695698 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:43.695813 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:43.695943 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.696066 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:43.696160 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.697418 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:43.697533 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:43.697816 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.697852 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:43.697967 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:43.698199 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:43.698478 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:43.698582 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:43.698665 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:43.698770 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.699297 [ 32281 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:43.699368 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:43.699452 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:43.699528 [ 32115 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:43.699535 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:43.700554 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:43.701550 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:43.701592 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:43.701660 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:43.701726 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:43.702005 [ 32115 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:43.702063 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:43.702134 [ 32115 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:43.704416 [ 32115 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 12ms, ratio: 0.025641028, points per minute: 1282.0513)
2026.05.06 08:49:43.704601 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:43.704714 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:43.705000 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:43.705368 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:43.706050 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:43.706266 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:43.707155 [ 32115 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:43.708522 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:43.708627 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:43.708724 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:43.708912 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.709040 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.709135 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:43.709194 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:43.709313 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:43.709873 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:43.709949 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:43.710031 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:43.710180 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.710296 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.710384 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:43.710438 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:43.710558 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:43.711194 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:43.711269 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:43.711355 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:43.711503 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.711660 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.711761 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:43.711821 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:43.711932 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:43.712567 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:43.712651 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:43.712741 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:43.712891 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.713007 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.713092 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:43.713151 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:43.713247 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:43.713820 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:43.713889 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:43.713966 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:43.714093 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.714200 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.714280 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:43.714330 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:43.714427 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:43.714934 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:43.714998 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:43.715077 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:43.715202 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.715300 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.715386 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:43.715432 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:43.715540 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:43.715944 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:43.716007 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:43.716079 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:43.716204 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.716302 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:43.716376 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:43.716421 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:43.716525 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:43.716817 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:43.717175 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:43.717385 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:43.717433 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:43.719134 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:43.719746 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.720027 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.720542 [ 31869 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:43.721016 [ 31869 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:43.721081 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:43.721352 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:43.722507 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:43.723649 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:43.724206 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:43.724292 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:43.724374 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:43.725784 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:43.726778 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:43.726858 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:43.728600 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:43.730453 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:43.730528 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:43.730628 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:43.730721 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:43.734332 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:43.734483 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:43.734865 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:43.735289 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:43.736233 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:43.736581 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:43.741008 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:43.741147 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:43.741307 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:43.741613 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.741821 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.741979 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:43.742073 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:43.742270 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:43.743178 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:43.743283 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:43.743400 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:43.743682 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.743870 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.744005 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:43.744088 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:43.744250 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:43.745139 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:43.745241 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:43.745356 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:43.745572 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.745737 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.745863 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:43.745940 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:43.746090 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:43.746875 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:43.746986 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:43.747097 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:43.747311 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.747470 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.747634 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:43.747712 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:43.747866 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:43.748756 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:43.748859 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:43.748973 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:43.749178 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.749343 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.749475 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:43.749576 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:43.749742 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:43.750545 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:43.750648 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:43.750767 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:43.750972 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.751133 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.751253 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:43.751335 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:43.751487 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:43.752146 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:43.752258 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:43.752369 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:43.752592 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.752754 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:43.752871 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:43.752944 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:43.753101 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:43.753553 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:43.754064 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:43.754271 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:43.754314 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:43.757301 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:43.757969 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.758280 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:43.758603 [ 32090 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:43.758745 [ 32090 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:44.139328 [ 32103 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 8774ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:44.144265 [ 32093 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:44.150316 [ 32093 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:44.159502 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:44.160641 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:44.184540 [ 32447 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:44.184696 [ 32447 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:44.185011 [ 32447 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007189259 sec. (139.096 rows/sec., 3.26 KiB/sec.)
2026.05.06 08:49:44.185133 [ 32447 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:44.189615 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:44.190286 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:44.197277 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:44.198022 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:44.209875 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.212715 [ 32278 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.213197 [ 32278 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:44.396351 [ 32087 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.396849 [ 32087 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.396915 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:44.397172 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.397329 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.397464 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.399718 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.399783 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.401334 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.401365 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.401662 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:44.401839 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:44.401949 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:44.402032 [ 31861 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.402573 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:44.402618 [ 31861 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.402633 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:44.402822 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:44.402934 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:44.402950 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.403063 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.403339 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.403501 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:44.404534 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.404972 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:44.404988 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.405613 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:44.405800 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:44.406121 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:44.406291 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.406646 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:44.406771 [ 31858 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.407235 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:44.407566 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.407601 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.407957 [ 31857 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.408012 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:44.408146 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:44.408259 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:44.408354 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.408642 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.409458 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.410321 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.411013 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.411870 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.412239 [ 31856 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.412583 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:44.412662 [ 31856 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.412921 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.412927 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.413201 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.413540 [ 31859 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:44.413663 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:44.413792 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:44.414660 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.414784 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.416117 [ 31859 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 709ms, ratio: 0.025641028, points per minute: 21.699036)
2026.05.06 08:49:44.416316 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:44.416381 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:44.416435 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:44.416478 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.416858 [ 31853 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.417138 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:44.417174 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.417227 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:44.417248 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:44.417340 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:44.417352 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:44.417447 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:44.417445 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.417930 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:44.419712 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:44.419770 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.420215 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:44.420559 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:44.420743 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:44.421050 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:44.421476 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:44.422427 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.422488 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:44.422660 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:44.422827 [ 31848 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.423003 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:44.423136 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:44.423188 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.423246 [ 31859 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:44.423470 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.425551 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.426704 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:44.427574 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.428010 [ 31846 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.428494 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.428879 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.428924 [ 31859 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:44.429042 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:44.429203 [ 31859 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:44.429457 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:44.429552 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:44.429673 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:44.429786 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:44.430849 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.431734 [ 31859 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:44.432716 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:44.433493 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:44.433610 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:44.433707 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:44.433842 [ 31859 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:44.434606 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:44.434785 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:44.435172 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:44.435737 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:44.436451 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:44.436781 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:44.437189 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:44.438734 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:44.438877 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:44.441779 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:44.441925 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:44.442076 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:44.442352 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.442584 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.442759 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:44.442856 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:44.442877 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:44.443082 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:44.444112 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:44.444259 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:44.444404 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:44.444688 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.444892 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.445049 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:44.445144 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:44.445303 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:44.445337 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:44.445373 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:44.445488 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:44.445614 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:44.446413 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:44.446554 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:44.446695 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:44.446943 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.447137 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.447293 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:44.447387 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:44.447589 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:44.448593 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:44.448717 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:44.448878 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:44.449142 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.449339 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.449493 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:44.449604 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:44.449806 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:44.450153 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:44.450326 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:44.450878 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:44.450921 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:44.451005 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:44.451147 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:44.451398 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.451441 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:44.451662 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.451820 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:44.451914 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:44.452108 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:44.452548 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:44.453025 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:44.453258 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:44.453384 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:44.453563 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:44.453815 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.454011 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.454244 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:44.454333 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:44.454567 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:44.455338 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:44.455458 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:44.455640 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:44.455890 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.456087 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:44.456340 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:44.456435 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:44.456639 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:44.457176 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:44.457555 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:44.457705 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:44.457859 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:44.457882 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:44.458119 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.458276 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:44.458327 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.458366 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:44.458482 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:44.458591 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:44.458785 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:44.459685 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:44.459813 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:44.459952 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:44.460205 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.460420 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.460588 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:44.460678 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:44.460869 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:44.461889 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:44.462010 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:44.462147 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:44.462394 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.462596 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:44.462603 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.462799 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:44.462893 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:44.463082 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:44.464052 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:44.464006 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.464177 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:44.464348 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:44.464666 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.464669 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.464905 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.465071 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:44.465129 [ 31843 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:44.465165 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:44.465428 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:44.466480 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:44.466624 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:44.466764 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:44.467020 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.467225 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.467376 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:44.467471 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:44.467713 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:44.468648 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:44.468770 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:44.468908 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:44.469159 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.469426 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.469614 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:44.469704 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:44.469899 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:44.470641 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:44.470760 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:44.470895 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:44.471145 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.471344 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:44.471492 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:44.471598 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:44.471847 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:44.472335 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:44.472979 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:44.473366 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:44.473452 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:44.476777 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:44.478041 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.478670 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.479027 [ 31839 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:44.479214 [ 31839 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:44.639655 [ 31835 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:44.645723 [ 31835 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:44.655914 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:44.657117 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:44.664809 [ 31836 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.665236 [ 31836 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.665257 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:44.665621 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.665625 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.666016 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.667991 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.668085 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.669957 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.670033 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.670239 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:44.670406 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:44.670538 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:44.670585 [ 31833 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.671044 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:44.671075 [ 31833 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.671381 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.671418 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.671463 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:44.671695 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:44.671791 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.671861 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:44.672702 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:44.673568 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.673739 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.674405 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:44.675230 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:44.675365 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.675488 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.675505 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:44.675803 [ 31831 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.676049 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:44.676128 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:44.676197 [ 31831 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.676471 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.676478 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.676541 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:44.676879 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.678199 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:44.678385 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:44.678545 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:44.679073 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.679094 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.679261 [ 31849 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 8513ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:44.681054 [ 32385 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:44.681182 [ 32385 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:44.681248 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.681336 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.681470 [ 32385 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006765499 sec. (147.809 rows/sec., 3.46 KiB/sec.)
2026.05.06 08:49:44.681575 [ 32385 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:44.681796 [ 31824 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.682070 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:44.682138 [ 31824 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.682446 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.682475 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.682801 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.684333 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.684598 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.685442 [ 31829 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:44.685588 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:44.685788 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:44.685947 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:44.686176 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:44.686341 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:44.686773 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:44.686886 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:44.686988 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:44.687193 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:44.688545 [ 31829 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 254ms, ratio: 0.025641028, points per minute: 60.56936)
2026.05.06 08:49:44.688787 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:44.688913 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:44.689507 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:44.689648 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:44.689745 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:44.689772 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:44.690380 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:44.691945 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:44.692021 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:44.692166 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:44.692731 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:44.692978 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:44.693340 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:44.693850 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:44.695191 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:44.695364 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:44.695494 [ 31829 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:44.696170 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:44.696282 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:44.697139 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:44.698350 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:44.698427 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:44.698562 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:44.698677 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:44.701996 [ 31829 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:44.702099 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:44.702217 [ 31829 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:44.703374 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:44.703571 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:44.704011 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:44.704544 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:44.705410 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:44.705562 [ 31829 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:49:44.705823 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:44.710338 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:44.710487 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:44.710656 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:44.710940 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.711156 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.711316 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:44.711412 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:44.711662 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:44.712872 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:44.713012 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:44.713164 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:44.713216 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.713437 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.713680 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.713842 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:44.713956 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:44.714178 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:44.715344 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:44.715476 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:44.715675 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:44.715937 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.716165 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.716361 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:44.716472 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:44.716703 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:44.717569 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.717811 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:44.717952 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:44.718106 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:44.718234 [ 32274 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:44.718378 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.718611 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.718781 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:44.718886 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:44.719091 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:44.720393 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:44.720552 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:44.720699 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:44.720959 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.721169 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.721323 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:44.721421 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:44.721635 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:44.722621 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:44.722745 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:44.722888 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:44.723140 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.723358 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.723552 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:44.723736 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:44.723956 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:44.724860 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:44.725001 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:44.725157 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:44.725413 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.725634 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:44.725783 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:44.725873 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:44.726065 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:44.726612 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:44.727284 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:44.727746 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:44.727843 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:44.732073 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:44.733385 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.734042 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.734431 [ 31822 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:44.734709 [ 31822 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:44.735016 [ 31822 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:44.735103 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:44.735504 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:44.737725 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:44.739590 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:44.740527 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:44.740638 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:44.740741 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:44.743544 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:44.745882 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:44.746044 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:44.750130 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:44.752893 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:44.753006 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:44.753141 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:44.753265 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:44.758278 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:44.758480 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:44.759032 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:44.759703 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:44.760936 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:44.761332 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:44.766249 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:44.766403 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:44.766575 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:44.766868 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.767110 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.767274 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:44.767374 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:44.767647 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:44.768630 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:44.768763 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:44.768911 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:44.769176 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.769382 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.769558 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:44.769655 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:44.769855 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:44.770975 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:44.771103 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:44.771248 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:44.771535 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.771769 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.771934 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:44.772032 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:44.772226 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:44.773331 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:44.773473 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:44.773642 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:44.773910 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.774117 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.774279 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:44.774378 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:44.774586 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:44.775761 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:44.775909 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:44.776056 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:44.776321 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.776544 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.776702 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:44.776799 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:44.777000 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:44.778003 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:44.778131 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:44.778276 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:44.778550 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.778768 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.778957 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:44.779052 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:44.779249 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:44.780098 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:44.780226 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:44.780366 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:44.780638 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.780838 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:44.780987 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:44.781078 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:44.781273 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:44.781832 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:44.782527 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:44.782964 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:44.783087 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:44.787827 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:44.789346 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.790069 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:44.790487 [ 31808 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:44.790702 [ 31808 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:45.078054 [ 31805 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1748ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:45.096497 [ 31640 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #5
2026.05.06 08:49:45.096607 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:49:45.097027 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:45.237321 [ 31817 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.237770 [ 31817 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.237864 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:45.238093 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:45.238290 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.238483 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.240916 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.241007 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.242601 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.242721 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.242905 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:45.243085 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:45.243196 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:45.243272 [ 31793 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.243866 [ 31793 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.243907 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:45.243993 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:45.244140 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:45.244191 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:45.244248 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.244316 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:45.244499 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.245150 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:45.246372 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.246552 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.247170 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:45.248033 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:45.248317 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.248321 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:45.248437 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.248795 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:45.248944 [ 31809 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.249343 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:45.249415 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:45.249422 [ 31809 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.249812 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.249842 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:45.250135 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.251028 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:45.251221 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:45.251367 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:45.251701 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.251874 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.253336 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.253413 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.253772 [ 31797 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.254156 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:45.254239 [ 31797 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.254545 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.254557 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:45.254873 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.256344 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.256795 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.258096 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.258457 [ 31787 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.258543 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.258869 [ 31798 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:45.259023 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:45.259084 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:45.259153 [ 31787 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.259254 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:45.259431 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.259462 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:45.259773 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.261091 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.261566 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.261875 [ 31798 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 557ms, ratio: 0.025641028, points per minute: 27.620497)
2026.05.06 08:49:45.262118 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:45.262253 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:45.262564 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:45.262973 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:45.263114 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.263117 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:45.263306 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:45.263342 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:45.263453 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:45.263543 [ 31800 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.263576 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:45.263865 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:45.264025 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:45.264128 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.265714 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:45.266060 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.266507 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:45.266562 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:45.266808 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:45.267181 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:45.267669 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:45.268610 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.268886 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:45.269036 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:45.269120 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:45.269313 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:45.269462 [ 31798 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:45.273300 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:45.275279 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:45.275360 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:45.275484 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:45.275665 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:45.277254 [ 31798 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:45.277391 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:45.277577 [ 31798 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:45.280460 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:45.280680 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:45.280896 [ 31798 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 19ms, ratio: 0.025641028, points per minute: 809.7167)
2026.05.06 08:49:45.281075 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:45.281652 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:45.282655 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:45.283054 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:45.284084 [ 31798 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:45.287678 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:45.287831 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:45.287990 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:45.288278 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.288495 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.288675 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:45.288773 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:45.288981 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:45.289928 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:45.290059 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:45.290202 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:45.290458 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.290701 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.290859 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:45.290953 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:45.291148 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:45.292261 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:45.292391 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:45.292552 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:45.292807 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.293003 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.293160 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:45.293256 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:45.293446 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:45.294491 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:45.294638 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:45.294777 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:45.295028 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.295237 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.295388 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:45.295481 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:45.295729 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:45.296815 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:45.296945 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:45.297091 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:45.297350 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.297573 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.297730 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:45.297826 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:45.298025 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:45.299008 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:45.299149 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:45.299294 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:45.299568 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.299803 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.299961 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:45.300052 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:45.300247 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:45.301012 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:45.301133 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:45.301270 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:45.301539 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.301741 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:45.301889 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:45.301979 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:45.302170 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:45.302730 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:45.303378 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:45.303845 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:45.303942 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:45.307720 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:45.309066 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.309743 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.310142 [ 31784 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:45.310394 [ 31784 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.310744 [ 31784 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:45.310788 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:45.311104 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.313259 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.314967 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:45.315957 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:45.316116 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:45.316227 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:45.319035 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:45.321376 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:45.321539 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:45.325532 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:45.327735 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:45.327817 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:45.327943 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:45.328061 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:45.333148 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:45.333392 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:45.333938 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:45.334613 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:45.335759 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:45.336151 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:45.340746 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:45.340898 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:45.341049 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:45.341333 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.341563 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.341723 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:45.341820 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:45.342026 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:45.343058 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:45.343192 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:45.343338 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:45.343668 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.343881 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.344055 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:45.344152 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:45.344357 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:45.345462 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:45.345608 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:45.345753 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:45.346012 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.346214 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.346372 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:45.346469 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:45.346685 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:45.347717 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:45.347852 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:45.347994 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:45.348255 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.348470 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.348646 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:45.348743 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:45.348966 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:45.350158 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:45.350293 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:45.350439 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:45.350723 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.350933 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.351089 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:45.351185 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:45.351384 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:45.352451 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:45.352616 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:45.352761 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:45.353015 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.353213 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.353364 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:45.353454 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:45.353663 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:45.354415 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:45.354589 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:45.354746 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:45.355005 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.355206 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.355355 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:45.355447 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:45.355687 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:45.356248 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:45.356941 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:45.357347 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:45.357440 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:45.361788 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:45.363143 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.363845 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.364268 [ 31802 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:45.364459 [ 31802 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:45.372329 [ 31776 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:45.378389 [ 31776 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:45.387922 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:45.389011 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:45.416032 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.416606 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.529456 [ 31774 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 5294ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:45.605344 [ 31772 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9934ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:45.812054 [ 31783 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.812542 [ 31783 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.812592 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:45.812988 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.812986 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:45.813402 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.815851 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.815932 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.817938 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.817997 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.818254 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:45.818451 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:45.818599 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:45.818687 [ 31763 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.819172 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:45.819226 [ 31763 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.819380 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:45.819488 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:45.819532 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:45.819546 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.819671 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:45.819936 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.820246 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:45.821483 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.821707 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:45.822018 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.822558 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:45.822794 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:45.823290 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:45.823597 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.823873 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:45.825058 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.825409 [ 31764 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.825597 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:45.825779 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:45.825816 [ 31764 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.825778 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:45.825938 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:45.826142 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.826166 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:45.826574 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.828279 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.828762 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.830285 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.830720 [ 31759 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.830960 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:45.831113 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:45.831449 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.832600 [ 31768 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:45.832742 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:45.832901 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:45.833994 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.835858 [ 31768 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 551ms, ratio: 0.025641028, points per minute: 27.921263)
2026.05.06 08:49:45.835994 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:45.836103 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:45.836229 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:45.836799 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:45.836820 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:45.836876 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:45.836947 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:45.836960 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:45.837091 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:45.837755 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:45.839379 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:45.839925 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:45.840244 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:45.840486 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:45.840903 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:45.841369 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:45.842563 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:45.842733 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:45.843025 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:45.843215 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:45.843353 [ 31768 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:45.847568 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:45.847668 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:45.847731 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:45.847825 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:45.847903 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:45.850312 [ 31768 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:45.850488 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:45.850667 [ 31768 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:45.851466 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:45.851599 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:45.851988 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:45.852561 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:45.853483 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:45.853799 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:45.854252 [ 31768 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 19ms, ratio: 0.025641028, points per minute: 809.7167)
2026.05.06 08:49:45.856372 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:45.856478 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:45.856600 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:45.856793 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.856935 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.857043 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:45.857109 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:45.857241 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:45.857884 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:45.857978 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:45.858068 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:45.858224 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.858349 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.858450 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:45.858520 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:45.858643 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:45.859367 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:45.859451 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:45.859553 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:45.859721 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.859845 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.859945 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:45.860008 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:45.860120 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:45.860777 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:45.860860 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:45.860948 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:45.861094 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.861207 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.861300 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:45.861360 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:45.861468 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:45.862141 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:45.862225 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:45.862316 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:45.862469 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.862611 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.862709 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:45.862781 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:45.862897 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:45.863507 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:45.863618 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:45.863713 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:45.863868 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.863989 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.864085 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:45.864142 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:45.864256 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:45.864743 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:45.864820 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:45.864905 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:45.865053 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.865173 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:49:45.865273 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:45.865330 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:45.865443 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:45.865810 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:45.866256 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:45.866532 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:45.866595 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:45.869413 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:45.870073 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.870431 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.870879 [ 31771 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:45.871260 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:45.871321 [ 31771 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:45.871654 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:45.873302 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:45.874994 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:45.875892 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:45.876013 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:45.876118 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:45.878931 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:45.881238 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:45.881388 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:45.885520 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:45.888487 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:45.888605 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:45.888776 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:45.888913 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:45.894167 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:45.894358 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:45.894931 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:45.895458 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:45.896499 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:45.896931 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:45.901844 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:45.902001 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:45.902169 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:45.902484 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.902720 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.902886 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:45.902986 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:45.903205 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:45.904235 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:45.904372 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:45.904533 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:45.904813 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.905022 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.905198 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:45.905293 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:45.905498 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:45.906668 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:45.906812 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:45.906957 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:45.907238 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.907443 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.907648 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:45.907750 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:45.907958 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:45.908993 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:45.909140 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:45.909282 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:45.909556 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.909758 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.909925 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:45.910021 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:45.910243 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:45.911354 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:45.911484 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:45.911675 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:45.911947 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.912154 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.912310 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:45.912406 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:45.912645 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:45.913648 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:45.913776 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:45.913927 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:45.914200 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.914432 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.914646 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:45.914764 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:45.915003 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:45.915867 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:45.915997 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:45.916149 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:45.916404 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.916623 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:45.916770 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:45.916876 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:45.917072 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:45.917629 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:45.918332 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:45.918761 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:45.918853 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:45.923656 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:45.925009 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.925266 [ 31747 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:45.925693 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.926101 [ 31770 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:45.926285 [ 31770 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:45.931281 [ 31747 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:45.940340 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:45.941360 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:45.964316 [ 32465 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:45.964457 [ 32465 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:45.964771 [ 32465 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006467758 sec. (154.613 rows/sec., 3.62 KiB/sec.)
2026.05.06 08:49:45.964866 [ 32465 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:45.969313 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:45.969996 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:45.977304 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:45.977986 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:45.990610 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.994059 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:45.994640 [ 32274 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:46.178949 [ 31746 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.179378 [ 31746 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.179486 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:46.179812 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.180074 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.180209 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.182829 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.182926 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.184387 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.184498 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.184587 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:46.184748 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:46.184850 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:46.184921 [ 31741 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.185373 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:46.185380 [ 31741 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.185609 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:46.185657 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.185671 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:46.185715 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.185762 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:46.185976 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.186081 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:46.187751 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:46.188013 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.188044 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.188646 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:46.188880 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:46.189181 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:46.189671 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:46.190638 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.190783 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.191086 [ 31736 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.191197 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:46.191361 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:46.191492 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:46.191530 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.191587 [ 31736 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.191918 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:46.191937 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.192182 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.193896 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.193980 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.195897 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.195955 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.196390 [ 31734 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.196803 [ 31734 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.196802 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.197065 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:46.197143 [ 31742 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:46.197160 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.197258 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:46.197395 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:46.197410 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.199013 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.199259 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.199816 [ 31742 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 345ms, ratio: 0.025641028, points per minute: 44.59309)
2026.05.06 08:49:46.200005 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:46.200125 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:46.200729 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:46.200730 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:46.200839 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:46.200946 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:46.201067 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.201471 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:46.201581 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:46.201694 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:46.201800 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:46.202880 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:46.203692 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:46.203879 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:46.204261 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:46.204630 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:46.204769 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:46.206156 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:46.206289 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:46.206398 [ 31742 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:46.206875 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:46.207018 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:46.210970 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:46.212459 [ 31742 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:46.212581 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:46.212696 [ 31742 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:46.213169 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:46.213248 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:46.213373 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:46.213493 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:46.215205 [ 31742 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:49:46.217416 [ 31742 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:46.218916 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:46.219162 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:46.219677 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:46.220203 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:46.221292 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:46.221725 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:46.226344 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:46.226492 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:46.226663 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:46.226952 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.227170 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.227332 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:46.227432 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:46.227721 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:46.228672 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:46.228801 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:46.228945 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:46.229205 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.229407 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.229577 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:46.229684 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:46.229879 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:46.230953 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:46.231080 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:46.231222 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:46.231473 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.231713 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.231875 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:46.231972 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:46.232164 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:46.233140 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:46.233265 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:46.233404 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:46.233672 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.233866 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.234046 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:46.234196 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:46.234464 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:46.236041 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:46.236180 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:46.236329 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:46.236608 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.236817 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.236974 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:46.237071 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:46.237271 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:46.238262 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:46.238389 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:46.238549 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:46.238805 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.239019 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.239170 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:46.239262 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:46.239453 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:46.240256 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:46.240382 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:46.240533 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:46.240784 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.240981 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:46.241128 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:46.241219 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:46.241411 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:46.241955 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:46.242644 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:46.243126 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:46.243222 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:46.247362 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:46.248892 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:46.249604 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:46.250184 [ 31729 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.250561 [ 31729 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:46.250673 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.251098 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.253221 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.254953 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:46.255777 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:46.255892 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:46.255995 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:46.257754 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:46.258971 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:46.259085 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:46.261386 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:46.264587 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:46.264681 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:46.264804 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:46.264919 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:46.269649 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:46.269834 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:46.270281 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:46.270862 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:46.272045 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:46.272443 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:46.278690 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:46.278901 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:46.279089 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:46.279433 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.279763 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.279972 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:46.280096 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:46.280349 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:46.281594 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:46.281768 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:46.281944 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:46.282263 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.282538 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.282758 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:46.282884 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:46.283123 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:46.284568 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:46.284732 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:46.284902 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:46.285214 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.285450 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.285677 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:46.285801 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:46.286030 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:46.287358 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:46.287545 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:46.287770 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:46.288095 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.288340 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.288555 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:46.288694 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:46.288930 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:46.290328 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:46.290499 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:46.290702 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:46.291021 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.291270 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.291479 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:46.291653 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:46.291904 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:46.293173 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:46.293337 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:46.293533 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:46.293861 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.294127 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.294338 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:46.294454 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:46.294719 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:46.295763 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:46.295958 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:46.296140 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:46.296467 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.296744 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.296931 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:46.297051 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:46.297301 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:46.297985 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:46.298903 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:46.299433 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:46.299578 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:46.306546 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:46.307584 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:46.308019 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:46.308359 [ 31726 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:46.308460 [ 31726 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:46.601957 [ 31725 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1524ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:46.724792 [ 31703 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:46.731198 [ 31703 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:46.740539 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:46.741606 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:46.763694 [ 32402 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:46.763835 [ 32402 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:46.764128 [ 32402 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006784579 sec. (147.393 rows/sec., 3.45 KiB/sec.)
2026.05.06 08:49:46.764215 [ 32402 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:46.768701 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:46.769398 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:46.770868 [ 31701 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.771257 [ 31701 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.771294 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:46.771684 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.771713 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.772137 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.773991 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.774118 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.776017 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.776174 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.776240 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:46.776383 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:46.776487 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:46.776614 [ 31695 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.776780 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:46.777053 [ 31695 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.777049 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:46.777118 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:46.777217 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:46.777321 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:46.777350 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.777417 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.777483 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:46.777698 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.777909 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:46.778856 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.779072 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:46.779101 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.779671 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:46.779864 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:46.780161 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:46.780292 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.780626 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:46.781447 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.781779 [ 31688 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.781819 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:46.782001 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:46.782108 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:46.782127 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:46.782121 [ 31688 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.782380 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.782494 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.782941 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.783805 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.784792 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.785447 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.786818 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.787207 [ 31913 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.787493 [ 31913 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.787549 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:46.787785 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.787926 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.788082 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.788245 [ 31694 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:46.788365 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:46.788509 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:46.790245 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.790323 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.790771 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:46.791160 [ 31694 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 574ms, ratio: 0.025641028, points per minute: 26.802467)
2026.05.06 08:49:46.791344 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:46.791458 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:46.792181 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:46.792283 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:46.792385 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:46.792395 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:46.792419 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.792844 [ 31685 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.792903 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:46.793200 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.793350 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:46.793467 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:46.793476 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.793600 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:46.794530 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:46.794867 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:46.795310 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:46.795341 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.795421 [ 32281 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:46.795529 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:46.795921 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:46.796358 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:46.796714 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:46.797487 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:46.797635 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.797661 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:46.797771 [ 31694 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:46.797961 [ 31900 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.798285 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.798559 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.799095 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:46.799261 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:46.800269 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.801806 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:46.802164 [ 31696 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:46.802466 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:46.802752 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:46.804572 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:46.804951 [ 31694 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:46.805054 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:46.805174 [ 31694 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:46.806425 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:46.806558 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:46.807472 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:46.807638 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:46.807751 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:46.807862 [ 31694 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:49:46.808848 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:46.808930 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:46.809054 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:46.809172 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:46.809910 [ 31694 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:46.810409 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:46.812684 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:46.812814 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:46.814242 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:46.814431 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:46.814854 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:46.815370 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:46.816487 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:46.816908 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:46.816995 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:46.819591 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:46.819706 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:46.819828 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:46.819944 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:46.821636 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:46.821790 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:46.821947 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:46.822241 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.822462 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.822644 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:46.822744 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:46.822957 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:46.823954 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:46.824090 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:46.824237 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:46.824548 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:46.824498 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.824739 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:46.824851 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.825030 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:46.825126 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:46.825203 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:46.825324 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:46.825707 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:46.826434 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:46.826585 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:46.826640 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:46.826729 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:46.826986 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.827015 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:46.827187 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.827349 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:46.827446 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:46.827693 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:46.828698 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:46.828826 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:46.828983 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:46.829236 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.829433 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.829607 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:46.829703 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:46.829893 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:46.830993 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:46.831124 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:46.831271 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:46.831550 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.831794 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.831884 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:46.831956 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:46.832034 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:46.832053 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:46.832188 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:46.832254 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:46.832482 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.832756 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.832935 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:46.833037 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:46.833248 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:46.833259 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:46.833431 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:46.833651 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:46.833918 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.834122 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.834239 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:46.834281 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:46.834374 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:46.834392 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:46.834544 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:46.834632 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:46.834805 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.835011 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.835169 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:46.835263 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:46.835436 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:46.835456 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:46.835586 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:46.835777 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:46.836042 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.836248 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:46.836415 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:46.836509 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:46.836656 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:46.836721 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:46.836793 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:46.836938 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:46.837192 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.837291 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:46.837390 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.837565 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:46.837679 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:46.837890 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:46.838053 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:46.838480 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:46.838592 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:46.838923 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:46.839067 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:46.839209 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:46.839458 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.839720 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.839880 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:46.839975 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:46.840165 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:46.841282 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:46.841411 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:46.841573 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:46.841848 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.842056 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.842212 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:46.842219 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:46.842309 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:46.842525 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:46.843545 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:46.843709 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:46.843854 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:46.843781 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:46.844109 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.844330 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.844484 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:46.844495 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:46.844592 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:46.844867 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:46.844924 [ 31865 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:46.845673 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:46.845795 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:46.845935 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:46.846187 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.846384 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:46.846547 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:46.846650 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:46.846840 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:46.847382 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:46.848097 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:46.848507 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:46.848617 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:46.852114 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:46.853628 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:46.854424 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:46.854882 [ 32154 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:46.855171 [ 32154 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:47.059924 [ 32068 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1531ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:47.230756 [ 32179 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.231174 [ 32179 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.231296 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:47.231749 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.231750 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:47.232259 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.234266 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.234455 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.235735 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.235909 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:47.236049 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:47.236153 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:47.236166 [ 31899 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.236532 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:47.236748 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.236883 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.236926 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:47.237067 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:47.237193 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:47.237902 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:47.238594 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.239296 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:47.239987 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:47.240195 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.240261 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:47.240585 [ 31684 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.240685 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:47.240809 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:47.240906 [ 31684 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.241058 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.241179 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:47.241186 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:47.241473 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.242359 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:47.242537 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:47.242653 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:47.243044 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.243268 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.244536 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.244847 [ 31873 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.245115 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.245273 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:47.245491 [ 32079 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.245846 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:47.245937 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.246108 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.247958 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.247971 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.248365 [ 32067 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:47.248484 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:47.248658 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:47.249811 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:47.249830 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.250600 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:47.250727 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:47.250828 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:47.250943 [ 32067 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 441ms, ratio: 0.025641028, points per minute: 34.88575)
2026.05.06 08:49:47.251117 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:47.251221 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:47.251860 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:47.251953 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:47.252046 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:47.252541 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:47.252563 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:47.253813 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:47.253826 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:47.253912 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:47.254456 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:47.254641 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:47.254932 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:47.255334 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:47.256110 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:47.256654 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:47.256791 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:47.256905 [ 32067 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:47.258626 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:47.258699 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:47.258817 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:47.258928 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:47.261910 [ 32067 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:47.262003 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:47.262114 [ 32067 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:47.263797 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:47.263981 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:47.264311 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:47.264614 [ 32067 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 13ms, ratio: 0.025641028, points per minute: 1183.432)
2026.05.06 08:49:47.264844 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:47.265719 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:47.266093 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:47.270666 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:47.270811 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:47.270961 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:47.271239 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.271454 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.271676 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:47.271779 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:47.271985 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:47.272940 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:47.273068 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:47.273207 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:47.273458 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.273674 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.273829 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:47.273922 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:47.274117 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:47.275182 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:47.275307 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:47.275444 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:47.275753 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.275956 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.276112 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:47.276209 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:47.276411 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:47.277388 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:47.277528 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:47.277666 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:47.277914 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.278109 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.278260 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:47.278351 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:47.278552 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:47.279658 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:47.279800 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:47.279944 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:47.280200 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.280406 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.280594 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:47.280688 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:47.280885 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:47.281852 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:47.281975 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:47.282119 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:47.282367 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.282576 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.282725 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:47.282814 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:47.283003 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:47.283779 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:47.283903 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:47.284037 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:47.284283 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.284479 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:47.284653 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:47.284743 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:47.284934 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:47.285454 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:47.286117 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:47.286527 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:47.286618 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:47.290721 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:47.292073 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.292717 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.293183 [ 31874 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.293583 [ 31874 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:47.293612 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:47.294007 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.296223 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.298037 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:47.298808 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:47.298915 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:47.299014 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:47.302485 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:47.305228 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:47.305398 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:47.309718 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:47.312021 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:47.312103 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:47.312239 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:47.312354 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:47.317063 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:47.317249 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:47.317783 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:47.318381 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:47.319398 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:47.319838 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:47.324696 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:47.324843 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:47.324992 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:47.325269 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.325483 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.325660 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:47.325757 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:47.325997 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:47.326986 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:47.327116 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:47.327277 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:47.327561 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.327796 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.327956 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:47.328051 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:47.328244 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:47.329340 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:47.329466 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:47.329620 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:47.329870 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.330065 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.330219 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:47.330311 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:47.330500 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:47.331501 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:47.331692 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:47.331835 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:47.332085 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.332280 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.332434 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:47.332541 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:47.332729 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:47.333812 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:47.333939 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:47.334083 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:47.334338 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.334560 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.334715 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:47.334810 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:47.335019 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:47.336040 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:47.336169 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:47.336313 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:47.336582 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.336780 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.336927 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:47.337015 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:47.337208 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:47.337965 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:47.338085 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:47.338220 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:47.338466 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.338677 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:47.338837 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:47.338927 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:47.339118 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:47.339740 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:47.340393 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:47.340813 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:47.340904 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:47.345300 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:47.346951 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.347854 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.348555 [ 31894 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:47.348770 [ 31894 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:47.369107 [ 31904 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 5545ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:47.504718 [ 31917 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:47.510899 [ 31917 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:47.516786 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:47.517506 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:47.532786 [ 32476 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:47.532942 [ 32476 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:47.533252 [ 32476 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.005323436 sec. (187.849 rows/sec., 4.40 KiB/sec.)
2026.05.06 08:49:47.533366 [ 32476 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:47.536880 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:47.537310 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:47.541505 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:47.542030 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:47.549359 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.551573 [ 32286 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.551977 [ 32286 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:47.714018 [ 32061 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:47.718009 [ 32061 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:47.727317 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:47.728438 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:47.750675 [ 32329 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:47.750814 [ 32329 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:47.751106 [ 32329 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006672209 sec. (149.875 rows/sec., 3.51 KiB/sec.)
2026.05.06 08:49:47.751192 [ 32329 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:47.755399 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:47.756080 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:47.762909 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:47.763573 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:47.774894 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.777849 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.778307 [ 32281 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:47.792327 [ 31927 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.792817 [ 31927 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.792889 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:47.793198 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:47.793282 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.793603 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.795716 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.795818 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.797639 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.797689 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.797929 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:47.798082 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:47.798192 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:47.798308 [ 32185 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.798694 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:47.798779 [ 32185 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.799032 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:47.799083 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.799175 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:47.799210 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:47.799319 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:47.799666 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.799999 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:47.801067 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.801396 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:47.801500 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.802068 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:47.802300 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:47.802649 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:47.802899 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.803149 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:47.803339 [ 32156 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.803629 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:47.803816 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.804291 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.804506 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:47.804674 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:47.804791 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:47.804853 [ 31925 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.805267 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:47.805388 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.805572 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.807054 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.807383 [ 32088 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.807597 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.807678 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:47.807858 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.809595 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.809643 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.811260 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.811633 [ 32092 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.811726 [ 31895 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:47.811864 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:47.811941 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:47.811991 [ 32092 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.812075 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:47.812250 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:47.812256 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.812550 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.814216 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.814266 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.814848 [ 31895 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 550ms, ratio: 0.025641028, points per minute: 27.972029)
2026.05.06 08:49:47.815108 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:47.815236 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:47.815846 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.815926 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:47.815951 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:47.816053 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:47.816185 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:47.816275 [ 31863 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.816659 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:47.816775 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:47.816863 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:47.816883 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:47.816897 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.816987 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:47.818356 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:47.818772 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.819088 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:47.819279 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:47.819732 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:47.819898 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:47.820297 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:47.822703 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:47.822823 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:47.822863 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:47.822982 [ 31895 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:47.823008 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:47.823223 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:47.823950 [ 32171 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:47.824447 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:47.824800 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:47.827930 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:47.829141 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:47.830179 [ 31895 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:47.830317 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:47.830453 [ 31895 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:47.830668 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:47.830759 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:47.830839 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:47.830964 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:47.831082 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:47.831278 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:47.831342 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:47.831400 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:47.832771 [ 31895 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 18ms, ratio: 0.025641028, points per minute: 854.70087)
2026.05.06 08:49:47.833130 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:47.834455 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:47.834610 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:47.834951 [ 31895 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:47.836046 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:47.836238 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:47.836804 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:47.837042 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:47.837487 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:47.838479 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:47.838895 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:47.840069 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:47.840158 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:47.840286 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:47.840407 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:47.843425 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:47.843703 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:47.843935 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:47.844343 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.844684 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.844870 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:47.844973 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:47.845202 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:47.845219 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:47.845411 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:47.845982 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:47.846176 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:47.846310 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:47.846456 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:47.846558 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:47.846743 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.846950 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.847106 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:47.847201 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:47.847403 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:47.847483 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:47.847945 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:47.848547 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:47.848677 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:47.848831 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:47.849087 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.849285 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.849442 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:47.849552 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:47.849745 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:47.850728 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:47.850852 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:47.850992 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:47.851245 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.851440 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.851642 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:47.851740 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:47.851930 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:47.852971 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:47.853038 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:47.853124 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:47.853168 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:47.853280 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:47.853313 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:47.853588 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.853610 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.853794 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.853835 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.853951 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:47.854000 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:47.854048 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:47.854101 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:47.854246 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:47.854312 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:47.855244 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:47.855308 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:47.855375 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:47.855468 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:47.855541 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:47.855676 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:47.855828 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.855948 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.856029 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.856153 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.856180 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:47.856271 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:47.856312 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:47.856405 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:47.856464 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:47.856636 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:47.857223 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:47.857344 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:47.857480 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:47.857748 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.857869 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:47.857944 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:47.858010 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:47.858091 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:47.858157 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:47.858181 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:47.858373 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:47.858416 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.858657 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.858818 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:47.858910 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:47.858912 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:47.859136 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:47.859945 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:47.860257 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:47.860371 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:47.860493 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:47.860549 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:47.860697 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:47.860728 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.860895 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.861047 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:47.861143 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:47.861307 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:47.862262 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:47.862370 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:47.862492 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:47.862724 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.862898 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.863028 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:47.863108 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:47.863271 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:47.864131 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:47.864241 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:47.864363 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:47.864588 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.864754 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.864882 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:47.864968 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:47.865084 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:47.865128 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:47.865769 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:47.865871 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:47.865985 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:47.866193 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.866358 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:47.866481 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:47.866570 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:47.866596 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.866729 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:47.867179 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:47.867296 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.867842 [ 32183 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:47.867864 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:47.868295 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:47.868375 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:47.872286 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:47.873499 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.874087 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:47.874486 [ 32165 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:47.874707 [ 32165 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:48.082240 [ 32108 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.082547 [ 32108 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.082690 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:48.082806 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:48.083088 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.083154 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.085688 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.085785 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.087443 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.087561 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.087742 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:48.087916 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:48.088026 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:48.088207 [ 32099 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.088697 [ 32099 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.088690 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:48.089025 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.089088 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:48.089156 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:48.089266 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:48.089374 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:48.089479 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.089974 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:48.091062 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.091466 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:48.091537 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.092178 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:48.092396 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:48.092713 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:48.092727 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.093204 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:48.094190 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.094464 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:48.094630 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:48.094645 [ 32196 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.094741 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:48.094993 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:48.095046 [ 32196 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.095289 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.095314 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:48.095579 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.097018 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.097146 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.098554 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.098689 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.099102 [ 32148 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.099481 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:48.099545 [ 32148 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.099823 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.099838 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:48.099905 [ 32126 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:48.100016 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:48.100141 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:48.100194 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.101722 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.101847 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.102344 [ 32126 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 267ms, ratio: 0.025641028, points per minute: 57.620285)
2026.05.06 08:49:48.102536 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:48.102631 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:48.103155 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:48.103247 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:48.103345 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:48.103479 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:48.103569 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.103896 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:48.104304 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:48.104414 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:48.104539 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:48.105181 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:48.105860 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:48.106041 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:48.106358 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:48.106891 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:48.107271 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:48.108242 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:48.108370 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:48.108475 [ 32126 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:48.109689 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:48.109834 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:48.114880 [ 32126 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:48.114980 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:48.115094 [ 32126 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:48.116337 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:48.117625 [ 32126 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:48.118851 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:48.118941 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:48.119063 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:48.119179 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:48.119688 [ 32126 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:48.124097 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:48.124282 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:48.124729 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:48.125288 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:48.126308 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:48.126738 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:48.131372 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:48.131537 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:48.131720 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:48.132004 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.132228 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.132385 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:48.132481 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:48.132698 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:48.133631 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:48.133757 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:48.133898 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:48.134158 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.134389 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.134560 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:48.134655 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:48.134848 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:48.135949 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:48.136078 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:48.136219 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:48.136467 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.136675 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.136839 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:48.136934 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:48.137123 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:48.138224 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:48.138359 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:48.138540 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:48.138800 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.139044 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.139246 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:48.139346 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:48.139560 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:48.140763 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:48.140939 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:48.141103 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:48.141375 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.141607 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.141769 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:48.141868 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:48.142066 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:48.143073 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:48.143198 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:48.143343 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:48.143648 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.143849 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.144003 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:48.144093 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:48.144286 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:48.145078 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:48.145201 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:48.145336 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:48.145600 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.145796 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:48.145942 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:48.146031 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:48.146235 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:48.146772 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:48.147445 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:48.147904 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:48.148001 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:48.151842 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:48.153407 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:48.154143 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:48.154614 [ 32094 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:48.154948 [ 32094 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.155264 [ 32094 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:48.155316 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:48.155734 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.157701 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.159333 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:48.160143 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:48.160257 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:48.160390 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:48.163217 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:48.165509 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:48.165677 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:48.169680 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:48.172163 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:48.172249 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:48.172377 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:48.172494 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:48.177560 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:48.177762 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:48.178277 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:48.178917 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:48.180046 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:48.180444 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:48.185266 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:48.185453 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:48.185643 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:48.185950 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.186169 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.186335 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:48.186437 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:48.186664 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:48.187721 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:48.187869 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:48.188021 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:48.188302 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.188549 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.188713 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:48.188810 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:48.189014 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:48.190129 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:48.190261 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:48.190404 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:48.190686 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.190887 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.191048 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:48.191145 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:48.191339 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:48.192537 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:48.192701 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:48.192891 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:48.193161 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.193362 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.193539 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:48.193637 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:48.193835 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:48.194948 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:48.195081 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:48.195228 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:48.195488 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.195742 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.195904 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:48.196002 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:48.196202 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:48.197202 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:48.197331 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:48.197475 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:48.197747 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.197948 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.198100 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:48.198191 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:48.198385 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:48.199157 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:48.199278 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:48.199414 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:48.199763 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.199965 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:48.200113 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:48.200206 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:48.200415 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:48.200970 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:48.201672 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:48.202093 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:48.202185 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:48.206035 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:48.206716 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:48.207037 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:48.207437 [ 32151 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:48.207689 [ 32151 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:48.305436 [ 32168 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1703ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:48.593172 [ 32175 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.593458 [ 32175 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.593702 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:48.593779 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:48.594138 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.594146 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.596720 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.596788 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.598445 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.598532 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.598677 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:48.598757 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:48.598817 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:48.599051 [ 32125 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.599570 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:48.599685 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:48.599598 [ 32125 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.599749 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:48.599979 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:48.599992 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:48.600098 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.600119 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.600531 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:48.601819 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.601832 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:48.601928 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.602664 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:48.602859 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:48.603272 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:48.603931 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:48.604665 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.604761 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.605135 [ 32139 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.605360 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:48.605438 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:48.605507 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:48.605593 [ 32139 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.605662 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:48.605778 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.606024 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:48.606402 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.608291 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.608777 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.610234 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.610630 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.610894 [ 32192 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.611165 [ 32192 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.611164 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:48.611325 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.611435 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:48.611847 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.612487 [ 31911 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:48.612634 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:48.612710 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.612787 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:48.613334 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.613997 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.614307 [ 32054 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.614687 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:48.614885 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:48.615107 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.615130 [ 31911 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 496ms, ratio: 0.025641028, points per minute: 31.017372)
2026.05.06 08:49:48.615339 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:48.615455 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:48.615573 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:48.615706 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:48.615810 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:48.616021 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:48.616119 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:48.616221 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:48.616741 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:48.616736 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.617340 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:48.618168 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:48.618400 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:48.618501 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:48.618544 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:48.618887 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:48.619091 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:48.619474 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:48.619916 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:48.620429 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:48.621194 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:48.621353 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:48.621473 [ 31911 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:48.622994 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:48.623068 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:48.623190 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:48.623304 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:48.627006 [ 31911 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:48.627123 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:48.627254 [ 31911 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:48.627880 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:48.628068 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:48.628466 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:48.628974 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:48.629809 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:48.630202 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:48.630453 [ 31911 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:48.632707 [ 31911 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:48.634730 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:48.634874 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:48.635025 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:48.635302 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.635531 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.635724 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:48.635823 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:48.636030 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:48.636964 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:48.637093 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:48.637234 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:48.637486 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.637700 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.637871 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:48.637964 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:48.638157 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:48.639215 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:48.639340 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:48.639479 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:48.639778 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.639976 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.640133 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:48.640226 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:48.640416 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:48.641379 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:48.641502 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:48.641658 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:48.641903 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.642109 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.642260 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:48.642351 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:48.642552 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:48.643691 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:48.643823 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:48.643964 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:48.644217 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.644418 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.644584 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:48.644677 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:48.644872 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:48.645833 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:48.645956 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:48.646097 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:48.646358 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.646567 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.646716 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:48.646805 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:48.646996 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:48.647794 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:48.647919 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:48.648056 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:48.648305 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.648505 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:48.648666 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:48.648757 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:48.648948 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:48.649473 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:48.650173 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:48.650599 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:48.650691 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:48.655045 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:48.656400 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:48.657139 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:48.657578 [ 32046 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:48.657942 [ 32046 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:48.658267 [ 32046 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:48.658330 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:48.658682 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:48.660803 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:48.662549 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:48.663332 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:48.663441 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:48.663565 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:48.666318 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:48.668631 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:48.668775 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:48.672754 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:48.675044 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:48.675137 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:48.675259 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:48.675375 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:48.680506 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:48.680710 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:48.681244 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:48.681865 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:48.683009 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:48.683268 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:48.685347 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:48.685433 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:48.685532 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:48.685689 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.685804 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.685889 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:48.685941 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:48.686052 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:48.686544 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:48.686614 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:48.686691 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:48.686822 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.686922 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.687007 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:48.687056 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:48.687151 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:48.687758 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:48.687827 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:48.687901 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:48.688028 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.688129 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.688212 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:48.688263 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:48.688357 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:48.688898 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:48.688967 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:48.689043 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:48.689169 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.689271 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.689348 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:48.689395 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:48.689488 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:48.690053 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:48.690120 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:48.690198 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:48.690325 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.690441 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.690536 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:48.690587 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:48.690686 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:48.691204 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:48.691274 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:48.691351 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:48.691488 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.691620 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.691719 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:48.691767 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:48.691865 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:48.692255 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:48.692316 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:48.692389 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:48.692541 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.692654 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:48.692739 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:48.692793 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:48.692908 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:48.693209 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:48.693574 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:48.693784 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:48.693831 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:48.697913 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:48.699037 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:48.699588 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:48.699989 [ 32043 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:48.700234 [ 32043 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:48.797834 [ 32040 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:48.804539 [ 32040 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:48.815226 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:48.816644 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:48.840732 [ 32365 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:48.840884 [ 32365 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:48.841208 [ 32365 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006964739 sec. (143.580 rows/sec., 3.37 KiB/sec.)
2026.05.06 08:49:48.841300 [ 32365 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:48.845669 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:48.846343 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:48.853579 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:48.854282 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:48.866419 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:48.869810 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:48.870406 [ 32283 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:49.124905 [ 32031 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.125318 [ 32031 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.125416 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:49.125722 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:49.125838 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.126094 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.128547 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.128633 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.130335 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.130406 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.130564 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:49.130706 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:49.130818 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:49.130826 [ 32034 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.131185 [ 32034 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.131303 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:49.131444 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:49.131469 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:49.131588 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:49.131719 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:49.131783 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.131851 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.132297 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:49.133538 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.133653 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:49.133704 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.134315 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:49.134532 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:49.134852 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:49.135375 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:49.136264 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.136452 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.136616 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:49.136753 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:49.136744 [ 32045 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.136864 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:49.137132 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:49.137200 [ 32045 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.137492 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.137579 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:49.137918 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.139279 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.139817 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.140874 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.141620 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.141947 [ 32027 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.142054 [ 32194 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:49.142152 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:49.142238 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:49.142273 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:49.142307 [ 32027 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.142493 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.142567 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:49.142812 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.144110 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.144359 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.144580 [ 32194 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 512ms, ratio: 0.025641028, points per minute: 30.048079)
2026.05.06 08:49:49.144762 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:49.144858 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:49.145392 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:49.145484 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:49.145571 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.145594 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:49.145899 [ 32018 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.146079 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:49.146251 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:49.146533 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.146713 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:49.147395 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:49.147461 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:49.147504 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:49.147657 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:49.148154 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:49.148178 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.148337 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:49.148617 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:49.149063 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:49.150316 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:49.150411 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:49.150445 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.150472 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:49.150594 [ 32194 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:49.153571 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:49.153721 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:49.156548 [ 32194 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:49.156651 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:49.156774 [ 32194 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:49.159112 [ 32194 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:49.159869 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:49.161066 [ 32194 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:49.162126 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:49.162203 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:49.162323 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:49.162437 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:49.167160 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:49.167341 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:49.167746 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:49.168293 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:49.169246 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:49.169663 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:49.175000 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:49.175175 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:49.175328 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:49.175698 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.175918 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.176078 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:49.176175 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:49.176380 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:49.177385 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:49.177537 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:49.177683 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:49.177939 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.178143 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.178299 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:49.178407 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:49.178611 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:49.179721 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:49.179850 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:49.179990 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:49.180300 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.180499 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.180671 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:49.180765 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:49.180953 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:49.182042 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:49.182167 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:49.182307 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:49.182613 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.182810 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.182980 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:49.183073 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:49.183291 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:49.184391 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:49.184538 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:49.184683 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:49.184937 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.185143 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.185296 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:49.185390 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:49.185598 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:49.186557 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:49.186690 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:49.186855 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:49.187113 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.187325 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.187481 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:49.187676 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:49.187888 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:49.188664 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:49.188703 [ 32020 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:49.188786 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:49.188924 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:49.189174 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.189369 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:49.189534 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:49.189625 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:49.189816 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:49.190340 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:49.191257 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:49.191718 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:49.191812 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:49.195640 [ 32020 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:49.196072 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:49.197432 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.198112 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.198571 [ 32024 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.198969 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:49.199004 [ 32024 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:49.199281 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.201121 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.202760 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:49.203526 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:49.203654 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:49.203754 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:49.204828 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:49.205508 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:49.205880 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:49.206761 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:49.206884 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:49.209087 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:49.211871 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:49.211948 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:49.212064 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:49.212177 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:49.217024 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:49.217211 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:49.217735 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:49.218366 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:49.219490 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:49.219927 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:49.224858 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:49.225009 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:49.225158 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:49.225437 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.225664 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.225820 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:49.225916 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:49.226120 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:49.227037 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:49.227163 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:49.227302 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:49.227569 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.227807 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.227978 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:49.228080 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:49.228158 [ 32321 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:49.228273 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:49.228314 [ 32321 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:49.228626 [ 32321 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006570208 sec. (152.202 rows/sec., 3.57 KiB/sec.)
2026.05.06 08:49:49.228729 [ 32321 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:49.229337 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:49.229462 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:49.229618 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:49.229867 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.230061 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.230214 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:49.230308 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:49.230495 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:49.231456 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:49.231598 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:49.231790 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:49.232039 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.232232 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.232384 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:49.232477 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:49.232678 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:49.233041 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:49.233737 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:49.233741 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:49.233915 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:49.234055 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:49.234312 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.234532 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.234688 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:49.234784 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:49.234994 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:49.235992 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:49.236117 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:49.236258 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:49.236505 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.236714 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.236861 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:49.236950 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:49.237142 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:49.237911 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:49.238030 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:49.238164 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:49.238422 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.238628 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:49.238786 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:49.238875 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:49.239064 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:49.239596 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:49.240339 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:49.240755 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:49.240845 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:49.240998 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:49.241714 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:49.244613 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:49.245921 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.246568 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.246937 [ 32033 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:49.254250 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.257964 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.258484 [ 32272 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:49.447339 [ 32015 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1142ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:49.630527 [ 32008 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 10470ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:49.650579 [ 32012 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.650982 [ 32012 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.651063 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:49.651350 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:49.651443 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.651752 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.653978 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.654243 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.656151 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.656423 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.656476 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:49.656638 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:49.656741 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:49.656877 [ 32013 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.657233 [ 32013 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.657244 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:49.657408 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:49.657525 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:49.657507 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:49.657546 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.657668 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:49.657930 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.658257 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:49.659703 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.659829 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:49.659870 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.660569 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:49.660788 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:49.661140 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:49.661531 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.661715 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:49.661952 [ 32023 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.662315 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:49.662605 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.662720 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.663092 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:49.663233 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:49.663347 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:49.664310 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.665921 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.666259 [ 32001 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.666605 [ 32001 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.666603 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:49.666896 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:49.666906 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.667250 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.668828 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.668857 [ 32004 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:49.668981 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:49.669093 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.669162 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:49.670755 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:49.670867 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:49.671691 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:49.671750 [ 32004 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 510ms, ratio: 0.025641028, points per minute: 30.165915)
2026.05.06 08:49:49.671807 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:49.671911 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:49.671971 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:49.672071 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:49.672741 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:49.672831 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:49.672924 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:49.673391 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:49.674831 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:49.675123 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:49.675555 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:49.675753 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:49.676091 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:49.676534 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:49.677571 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:49.677717 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:49.677971 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:49.678104 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:49.678210 [ 32004 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:49.682040 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:49.683458 [ 32004 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:49.683573 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:49.683729 [ 32004 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:49.684722 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:49.684802 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:49.684943 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:49.685067 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:49.686302 [ 32004 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:49.690018 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:49.690224 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:49.690692 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:49.691247 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:49.692349 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:49.692864 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:49.698060 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:49.698224 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:49.698380 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:49.698687 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.698905 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.699078 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:49.699184 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:49.699393 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:49.700419 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:49.700582 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:49.700726 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:49.700985 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.701194 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.701353 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:49.701449 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:49.701662 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:49.702845 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:49.702983 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:49.703141 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:49.703427 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.703679 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.703863 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:49.703989 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:49.704204 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:49.705256 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:49.705386 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:49.705548 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:49.705803 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.705999 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.706166 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:49.706260 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:49.706448 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:49.707737 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:49.707896 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:49.708062 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:49.708336 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.708561 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.708720 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:49.708832 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:49.709030 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:49.710048 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:49.710197 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:49.710363 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:49.710642 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.710844 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.710997 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:49.711100 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:49.711313 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:49.712145 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:49.712273 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:49.712414 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:49.712691 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.712902 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:49.713092 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:49.713186 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:49.713380 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:49.713943 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:49.714664 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:49.715098 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:49.715209 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:49.719592 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:49.721002 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.721706 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.722229 [ 32009 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:49.722626 [ 32009 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:49.722658 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:49.722998 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:49.724952 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:49.726753 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:49.727480 [ 32029 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:49.727772 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:49.727881 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:49.727984 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:49.730794 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:49.733126 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:49.733235 [ 32029 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:49.733278 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:49.737350 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:49.739798 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:49.739895 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:49.740015 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:49.740130 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:49.743418 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:49.744607 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:49.744932 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:49.745117 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:49.745628 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:49.746238 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:49.747406 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:49.747836 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:49.752637 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:49.752785 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:49.752936 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:49.753216 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.753432 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.753610 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:49.753708 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:49.753913 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:49.754910 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:49.755050 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:49.755197 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:49.755465 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.755736 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.755898 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:49.755995 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:49.756193 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:49.757290 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:49.757420 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:49.757574 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:49.757825 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.758022 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.758180 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:49.758276 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:49.758481 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:49.759473 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:49.759644 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:49.759786 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:49.760034 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.760228 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.760383 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:49.760477 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:49.760683 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:49.761756 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:49.761884 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:49.762027 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:49.762283 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.762489 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.762672 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:49.762767 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:49.762961 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:49.763972 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:49.764104 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:49.764244 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:49.764494 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.764706 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.764858 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:49.764965 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:49.765169 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:49.766018 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:49.766150 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:49.766293 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:49.766566 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.766781 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:49.766930 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:49.767089 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:49.767286 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:49.767879 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:49.768639 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:49.768916 [ 32458 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:49.769041 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:49.769050 [ 32458 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:49.769132 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:49.769344 [ 32458 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007069309 sec. (141.457 rows/sec., 3.32 KiB/sec.)
2026.05.06 08:49:49.769432 [ 32458 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:49.773258 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:49.773733 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:49.774432 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:49.774739 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.775459 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.775959 [ 31993 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:49.776177 [ 31993 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:49.782088 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:49.782790 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:49.795465 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.798825 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:49.799385 [ 32281 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:50.023797 [ 31984 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.024295 [ 31984 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.024293 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:50.024754 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.024770 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:50.025180 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.027420 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.027498 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.029197 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.029295 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.029454 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:50.029621 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:50.029724 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:50.029829 [ 31988 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.030249 [ 31988 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.030289 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:50.030357 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:50.030463 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:50.030619 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:50.030604 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:50.030677 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.031010 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.031258 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:50.032795 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:50.032913 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.033015 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.033558 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:50.033786 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:50.034115 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:50.034652 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:50.035762 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.036034 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.036092 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:50.036273 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:50.036393 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:50.036481 [ 31996 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.036887 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:50.036925 [ 31996 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.037192 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:50.037216 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.037468 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.039401 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.039536 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.041394 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.041627 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.041933 [ 31968 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.042254 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:50.042272 [ 31981 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:50.042391 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:50.042438 [ 31968 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.042581 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:50.042691 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.042701 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:50.042978 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.044610 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.044849 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.045178 [ 31981 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 359ms, ratio: 0.025641028, points per minute: 42.854084)
2026.05.06 08:49:50.045373 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:50.045478 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:50.046283 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:50.046392 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:50.046475 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:50.046495 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:50.046947 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.047084 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:50.047337 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:50.047453 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:50.047576 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:50.048841 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:50.049596 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:50.049659 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:50.049844 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:50.050234 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:50.050740 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:50.051053 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:50.051164 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:50.052269 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:50.052406 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:50.052510 [ 31981 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:50.053664 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:50.056423 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:50.056500 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:50.056637 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:50.056749 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:50.057652 [ 31981 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:50.057742 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:50.057848 [ 31981 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:50.060690 [ 31981 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:50.061483 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:50.061683 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:50.061993 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:50.062560 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:50.063743 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:50.063897 [ 31981 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 4ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:50.064147 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:50.068860 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:50.069014 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:50.069169 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:50.069459 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.069693 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.069855 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:50.069953 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:50.070162 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:50.071111 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:50.071239 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:50.071380 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:50.071696 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.071902 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.072058 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:50.072167 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:50.072362 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:50.073465 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:50.073608 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:50.073747 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:50.073997 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.074196 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.074351 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:50.074446 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:50.074651 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:50.075669 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:50.075807 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:50.075945 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:50.076195 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.076388 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.076569 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:50.076661 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:50.076847 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:50.077922 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:50.078049 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:50.078210 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:50.078561 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.078785 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.078947 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:50.079044 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:50.079243 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:50.080273 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:50.080400 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:50.080562 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:50.080815 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.081026 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.081175 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:50.081264 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:50.081455 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:50.082211 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:50.082329 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:50.082461 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:50.082728 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.082924 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:50.083069 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:50.083158 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:50.083347 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:50.083929 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:50.084609 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:50.085019 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:50.085110 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:50.089989 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:50.091427 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:50.092180 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:50.092643 [ 31963 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:50.092916 [ 31963 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.093272 [ 31963 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:50.093322 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:50.093686 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.095660 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.097623 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:50.098470 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:50.098594 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:50.098698 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:50.101494 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:50.103811 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:50.103961 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:50.107967 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:50.110312 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:50.110392 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:50.110527 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:50.110643 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:50.115375 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:50.115577 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:50.116092 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:50.116683 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:50.117722 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:50.118136 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:50.122922 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:50.123074 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:50.123230 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:50.123535 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.123796 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.123964 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:50.124064 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:50.124275 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:50.125242 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:50.125374 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:50.125542 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:50.125808 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.126011 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.126181 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:50.126290 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:50.126484 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:50.127638 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:50.127774 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:50.127941 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:50.128196 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.128392 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.128567 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:50.128663 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:50.128853 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:50.129850 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:50.129979 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:50.130120 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:50.130371 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.130582 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.130751 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:50.130848 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:50.131036 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:50.132193 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:50.132337 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:50.132483 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:50.132766 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.132975 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.133148 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:50.133246 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:50.133445 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:50.134441 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:50.134589 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:50.134735 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:50.134990 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.135205 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.135358 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:50.135448 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:50.135695 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:50.136460 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:50.136603 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:50.136740 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:50.136991 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.137188 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:50.137335 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:50.137425 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:50.137631 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:50.138159 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:50.138830 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:50.139247 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:50.139339 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:50.143916 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:50.145394 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:50.146112 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:50.146606 [ 31961 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:50.146812 [ 31961 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:50.409620 [ 31959 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:50.416254 [ 31959 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.69 GiB.
2026.05.06 08:49:50.425596 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:50.426859 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:50.452125 [ 32418 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:50.452225 [ 32418 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:50.452401 [ 32418 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007009149 sec. (142.671 rows/sec., 3.34 KiB/sec.)
2026.05.06 08:49:50.452453 [ 32418 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:50.456334 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:50.457000 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:50.463944 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:50.464638 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:50.476418 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:50.479313 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:50.479840 [ 32284 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:50.647706 [ 32315 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:50.648212 [ 32315 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:50 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE73A5650A494
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:50.648631 [ 32315 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:50.785089 [ 31948 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1338ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:50.803415 [ 31956 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.803886 [ 31956 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.803986 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:50.804237 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:50.804417 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.804626 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.806795 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.806856 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.808895 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.808980 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.809174 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:50.809357 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:50.809461 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:50.809507 [ 31945 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.809868 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:50.809946 [ 31945 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.810205 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.810209 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:50.810310 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:50.810358 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:50.810496 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:50.810718 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.811117 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:50.812252 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.812718 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:50.813415 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.813626 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:50.813833 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:50.814192 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:50.814724 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:50.815895 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.816271 [ 31942 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.816364 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:50.816563 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:50.816644 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.816669 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:50.816692 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:50.817046 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.818972 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.820737 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.821165 [ 31943 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.821544 [ 31943 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.821574 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:50.821797 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:50.821895 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.822068 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.822551 [ 31953 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:50.822654 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:50.822779 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:50.823866 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.823923 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.825234 [ 31953 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 761ms, ratio: 0.025641028, points per minute: 20.216316)
2026.05.06 08:49:50.825467 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:50.825585 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:50.825685 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:50.825787 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:50.826197 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:50.826296 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:50.826397 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:50.826616 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:50.826724 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:50.826826 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:50.826919 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:50.828414 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:50.829113 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:50.829302 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:50.829624 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:50.829700 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:50.830116 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:50.831464 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:50.831651 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:50.831764 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:50.831999 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:50.832169 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:50.836173 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:50.837683 [ 31953 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:50.837780 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:50.837893 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:50.838592 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:50.838672 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:50.838793 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:50.838906 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:50.840358 [ 31953 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:50.842785 [ 31953 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:50.843691 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:50.843879 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:50.844301 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:50.844803 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:50.845763 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:50.846192 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:50.850894 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:50.851040 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:50.851187 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:50.851469 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.851740 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.851903 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:50.852037 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:50.852241 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:50.853256 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:50.853390 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:50.853555 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:50.853814 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.854017 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.854171 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:50.854263 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:50.854453 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:50.855510 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:50.855695 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:50.855839 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:50.856102 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.856299 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.856454 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:50.856563 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:50.856750 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:50.857714 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:50.857837 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:50.857973 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:50.858216 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.858406 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.858573 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:50.858664 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:50.858848 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:50.859971 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:50.860103 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:50.860244 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:50.860498 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.860721 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.860875 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:50.860970 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:50.861163 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:50.862120 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:50.862242 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:50.862382 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:50.862651 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.862849 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.863034 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:50.863126 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:50.863318 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:50.864133 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:50.864257 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:50.864394 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:50.864662 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.864859 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:50.865005 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:50.865093 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:50.865282 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:50.865816 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:50.866454 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:50.866879 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:50.866971 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:50.871592 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:50.872915 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:50.873601 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:50.874108 [ 31926 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:50.874492 [ 31926 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:50.874546 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:50.874894 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:50.876992 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:50.878980 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:50.879909 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:50.880064 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:50.880175 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:50.882959 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:50.885293 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:50.885449 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:50.889474 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:50.891841 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:50.891926 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:50.892049 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:50.892167 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:50.897068 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:50.897261 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:50.897770 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:50.898330 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:50.899590 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:50.900010 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:50.904612 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:50.904785 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:50.904937 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:50.905220 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.905436 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.905628 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:50.905727 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:50.905936 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:50.906879 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:50.907008 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:50.907167 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:50.907425 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.907683 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.907845 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:50.907938 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:50.908133 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:50.909232 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:50.909360 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:50.909500 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:50.909766 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.909978 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.910135 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:50.910231 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:50.910421 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:50.911396 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:50.911537 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:50.911722 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:50.911974 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.912169 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.912320 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:50.912413 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:50.912618 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:50.913687 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:50.913812 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:50.913967 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:50.914232 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.914436 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.914610 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:50.914708 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:50.914906 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:50.915928 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:50.916059 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:50.916203 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:50.916453 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.916667 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.916819 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:50.916923 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:50.917116 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:50.917920 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:50.918044 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:50.918180 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:50.918427 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.918642 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:50.918788 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:50.918878 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:50.919069 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:50.919653 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:50.920326 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:50.920752 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:50.920844 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:50.925260 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:50.926591 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:50.927232 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:50.927686 [ 31922 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:50.927893 [ 31922 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:51.094487 [ 32128 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:51.094966 [ 32128 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:51.095018 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:51.095240 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:51.095412 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:51.095490 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:51.098108 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:51.098297 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:51.099984 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:51.100149 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:51.100190 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:51.100340 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:51.100685 [ 31930 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:51.100982 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:51.101041 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:51.101075 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:51.101098 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:51.101176 [ 31930 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:51.101428 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:51.101490 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:51.101509 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:51.101869 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:51.102724 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:51.103198 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:51.103310 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:51.103409 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:51.103565 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:51.103814 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:51.103950 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:51.105039 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:51.105120 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:51.105243 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:51.105714 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:51.105831 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:51.106187 [ 31933 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:51.106481 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:51.106568 [ 31933 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:51.106865 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:51.106895 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:51.107180 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:51.108887 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:51.109175 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:51.110774 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:51.110825 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:51.111143 [ 31919 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:51.111259 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:51.111384 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:51.111405 [ 31909 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:51.111792 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:51.111850 [ 31909 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:51.112116 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:51.112135 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:51.112428 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:51.113752 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:51.113881 [ 31919 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 272ms, ratio: 0.025641028, points per minute: 56.56109)
2026.05.06 08:49:51.114085 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:51.114100 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:51.114201 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:51.114777 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:51.114898 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:51.115011 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:51.115359 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:51.115577 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:51.115871 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:51.116197 [ 31905 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:51.116222 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:51.116333 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:51.116438 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:51.116530 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:51.116840 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:51.117339 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:51.118098 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:51.118319 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:51.118684 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:51.118692 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:51.119183 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:51.119344 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:51.120626 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:51.120803 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:51.120933 [ 31919 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:51.121247 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:51.122066 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:51.122244 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:51.126932 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:51.129022 [ 31919 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:51.129125 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:51.129181 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:51.129249 [ 31919 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:51.129264 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:51.129409 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:51.129554 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:51.131883 [ 31919 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 18ms, ratio: 0.025641028, points per minute: 854.70087)
2026.05.06 08:49:51.133918 [ 31919 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:51.134529 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:51.134717 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:51.135203 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:51.135791 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:51.136745 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:51.137146 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:51.141700 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:51.141869 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:51.142023 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:51.142309 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.142543 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.142705 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:51.142802 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:51.143030 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:51.144053 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:51.144189 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:51.144335 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:51.144606 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.144811 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.144966 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:51.145058 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:51.145250 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:51.146343 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:51.146469 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:51.146627 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:51.146910 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.147114 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.147273 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:51.147368 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:51.147669 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:51.148666 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:51.148810 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:51.148950 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:51.149198 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.149391 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.149560 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:51.149652 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:51.149839 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:51.150925 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:51.151052 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:51.151193 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:51.151444 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.151696 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.151869 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:51.151965 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:51.152161 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:51.153194 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:51.153331 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:51.153476 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:51.153757 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.153962 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.154113 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:51.154203 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:51.154394 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:51.155234 [ 32274 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:51.155391 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:51.155585 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:51.155898 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.156116 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32274, kind: Regular, unbound: 0
2026.05.06 08:49:51.156265 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:51.156356 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:51.156578 [ 32274 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:51.157197 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:51.157958 [ 32274 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:51.158437 [ 32274 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:51.158554 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:51.165319 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:51.166879 [ 32274 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:51.167732 [ 32274 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:51.168180 [ 31901 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:51.168486 [ 31901 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:51.168640 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:51.169004 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:51.170668 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:51.172046 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:51.172624 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:51.172685 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:51.172742 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:51.175470 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:51.177821 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:51.177974 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:51.182898 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:51.184836 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:51.184929 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:51.185051 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:51.185165 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:51.190045 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:51.190229 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:51.190801 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:51.191289 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:51.192497 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:51.192891 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:51.197439 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:51.197598 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:51.197749 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:51.198027 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.198257 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.198417 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:51.198534 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:51.198741 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:51.199708 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:51.199854 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:51.199996 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:51.200252 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.200451 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.200620 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:51.200714 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:51.200905 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:51.201969 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:51.202108 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:51.202282 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:51.202564 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.202761 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.202917 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:51.203013 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:51.203203 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:51.204224 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:51.204349 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:51.204486 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:51.204749 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.204941 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.205093 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:51.205185 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:51.205371 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:51.206428 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:51.206570 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:51.206711 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:51.206960 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.207160 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.207313 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:51.207420 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:51.207681 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:51.208650 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:51.208772 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:51.208912 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:51.209159 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.209353 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.209502 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:51.209607 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:51.209797 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:51.210535 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:51.210652 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:51.210784 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:51.211026 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.211219 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:51.211376 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:51.211465 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:51.211748 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:51.212261 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:51.212908 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:51.213303 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:51.213393 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:51.217301 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:51.218621 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:51.219252 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:51.219808 [ 32114 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:51.220053 [ 32114 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:51.436145 [ 31891 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 4067ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:51.503943 [ 31893 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:51.509751 [ 31893 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:51.521340 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:51.522363 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:51.540468 [ 32379 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:51.540636 [ 32379 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:51.540981 [ 32379 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.005280887 sec. (189.362 rows/sec., 4.44 KiB/sec.)
2026.05.06 08:49:51.541087 [ 32379 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:51.546129 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:51.546784 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:51.553558 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:51.554153 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:51.562406 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:51.564523 [ 32283 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:51.564842 [ 32283 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:51.814935 [ 31882 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 7136ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:51.996240 [ 31878 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:52.002254 [ 31878 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:52.012789 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:52.014261 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:52.038786 [ 32436 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:52.038910 [ 32436 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:52.039151 [ 32436 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006971818 sec. (143.435 rows/sec., 3.36 KiB/sec.)
2026.05.06 08:49:52.039217 [ 32436 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:52.043533 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:52.044205 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:52.051083 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:52.051935 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:52.063385 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:52.066209 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:52.066707 [ 32285 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:52.070481 [ 31876 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.070762 [ 31876 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.070870 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:52.071064 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:52.071253 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.071479 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.073568 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.073619 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.075266 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:52.075314 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:52.075571 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:52.075768 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:52.075880 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:52.075962 [ 31869 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.076345 [ 31869 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.076369 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:52.076496 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:52.076630 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:52.076655 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:52.076744 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.076775 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:52.076989 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.077357 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:52.078397 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.078409 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.078479 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:52.079057 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:52.079269 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:52.079675 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:52.079735 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:52.080079 [ 32090 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.080226 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:52.080423 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:52.080753 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.081129 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:52.081345 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:52.081478 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:52.081604 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:52.082154 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.083504 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:52.083975 [ 31888 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.084387 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:52.084400 [ 31888 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.084769 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.084795 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:52.085055 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.086286 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.086346 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.086722 [ 32100 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:52.086822 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:52.086972 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:52.087873 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:52.087924 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:52.088539 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:52.088647 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:52.088748 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:52.088994 [ 32100 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 955ms, ratio: 0.025641028, points per minute: 16.109547)
2026.05.06 08:49:52.089177 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:52.089269 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:52.089846 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:52.089962 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:52.090080 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:52.090622 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:52.091485 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:52.092070 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:52.092762 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:52.092960 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:52.093306 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:52.093794 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:52.093867 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:52.093938 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:52.095048 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:52.095173 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:52.095279 [ 32100 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:52.097931 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:52.100528 [ 32100 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:52.100584 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:52.100644 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:52.100672 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:52.100766 [ 32100 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:52.100804 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:52.100929 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:52.103100 [ 32100 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:52.105500 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:52.105731 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:52.106208 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:52.106862 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:52.108025 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:52.108429 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:52.113321 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:52.113466 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:52.113642 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:52.113927 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.114200 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.114392 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:52.114503 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:52.114760 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:52.115525 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:52.115599 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:52.115696 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:52.115838 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.115945 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.116036 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:52.116095 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:52.116201 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:52.116795 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:52.116863 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:52.116950 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:52.117084 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.117187 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.117271 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:52.117320 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:52.117416 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:52.117931 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:52.117997 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:52.118072 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:52.118197 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.118294 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.118377 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:52.118426 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:52.118540 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:52.119101 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:52.119168 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:52.119245 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:52.119375 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.119477 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.119567 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:52.119631 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:52.119741 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:52.120234 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:52.120298 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:52.120372 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:52.120499 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.120616 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.120694 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:52.120740 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:52.120834 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:52.121207 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:52.121267 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:52.121338 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:52.121462 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.121565 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:52.121637 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:52.121682 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:52.121775 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:52.122061 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:52.122382 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:52.122602 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:52.122649 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:52.125941 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:52.126535 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:52.126821 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:52.127271 [ 32081 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.127731 [ 32081 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:52.127788 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:52.128169 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.130055 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.131698 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:52.132612 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:52.132737 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:52.132845 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:52.135745 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:52.137996 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:52.138149 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:52.142326 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:52.144569 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:52.144651 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:52.144776 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:52.144894 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:52.149899 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:52.150093 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:52.150676 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:52.151281 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:52.152484 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:52.152922 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:52.157627 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:52.157785 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:52.157942 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:52.158232 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.158451 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.158633 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:52.158734 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:52.158945 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:52.159939 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:52.160076 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:52.160223 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:52.160487 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.160713 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.160872 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:52.160969 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:52.161168 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:52.162293 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:52.162426 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:52.162593 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:52.162856 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.163058 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.163230 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:52.163346 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:52.163564 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:52.164652 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:52.164787 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:52.164931 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:52.165237 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.165475 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.165660 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:52.165774 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:52.165971 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:52.167106 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:52.167244 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:52.167416 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:52.167779 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.168005 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.168172 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:52.168273 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:52.168476 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:52.169579 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:52.169717 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:52.169867 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:52.170133 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.170339 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.170510 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:52.170620 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:52.170818 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:52.171640 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:52.171770 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:52.171914 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:52.172174 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.172376 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:49:52.172571 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:52.172666 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:52.172865 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:52.173462 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:52.174214 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:52.174701 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:52.174816 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:52.178906 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:52.180290 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:52.180946 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:52.181386 [ 32087 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:52.181675 [ 32087 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:52.456261 [ 31861 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1671ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:52.918280 [ 32086 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.918727 [ 32086 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.918858 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:52.919031 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:52.919361 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.919377 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.922264 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.922370 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.924139 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:52.924277 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:52.924448 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:52.924623 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:52.924705 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:52.924751 [ 31853 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.925231 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:52.925279 [ 31853 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.925286 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:52.925396 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:52.925496 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:52.925597 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.925608 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:52.925936 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:52.925993 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.927255 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:52.927676 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.927755 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.927885 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:52.928018 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:52.928380 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:52.928877 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:52.930096 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:52.930189 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:52.930256 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:52.930333 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:52.930331 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:52.930925 [ 31846 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.931279 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:52.931334 [ 31846 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.931664 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.931667 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:52.931862 [ 31845 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:52.932017 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.933299 [ 31851 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:52.933369 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:52.933479 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:52.933547 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.933674 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.935155 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:52.935338 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:52.935441 [ 31851 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 832ms, ratio: 0.025641028, points per minute: 18.491125)
2026.05.06 08:49:52.935583 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:52.935670 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:52.935963 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:52.936094 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:52.936177 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:52.936231 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:52.936244 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:52.936341 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:52.936666 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:52.937806 [ 31845 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:52.937904 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:52.938551 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:52.938662 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:52.938939 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:52.938965 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:52.939498 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:52.940688 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:52.940777 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:52.940841 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:52.941236 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:52.941378 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:52.944862 [ 31851 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:52.944927 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:52.945006 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:52.946362 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:52.946653 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:52.947115 [ 31851 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 12ms, ratio: 0.025641028, points per minute: 1282.0513)
2026.05.06 08:49:52.947777 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:52.948748 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:52.948828 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:52.948950 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:52.949066 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:52.954351 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:52.954618 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:52.955150 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:52.955761 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:52.956677 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:52.957077 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:52.961708 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:52.961863 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:52.962023 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:52.962312 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.962558 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.962728 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:52.962828 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:52.963045 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:52.964041 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:52.964177 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:52.964326 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:52.964611 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.964826 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.965002 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:52.965099 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:52.965303 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:52.966408 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:52.966556 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:52.966704 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:52.966965 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.967192 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.967357 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:52.967468 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:52.967717 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:52.968882 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:52.969074 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:52.969286 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:52.969665 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:52.969694 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.969797 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:52.969974 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.970112 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006506988 sec. (153.681 rows/sec., 3.60 KiB/sec.)
2026.05.06 08:49:52.970144 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:52.970200 [ 32374 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:52.970243 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:52.970446 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:52.971588 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:52.971761 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:52.971915 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:52.972185 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.972408 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.972589 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:52.972694 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:52.972910 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:52.973938 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:52.974067 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:52.974216 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:52.974256 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:52.974486 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.974712 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.974869 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:52.974908 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:52.974962 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:52.975162 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:52.975966 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:52.976094 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:52.976235 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:52.976496 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.976731 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:52.976941 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:52.977040 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:52.977243 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:52.977997 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:52.978744 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:52.979163 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:52.979256 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:52.981856 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:52.982377 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:52.982459 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:52.983779 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:52.984461 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:52.984862 [ 31835 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:52.985168 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:52.985192 [ 31835 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:52.985573 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:52.987126 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:52.988637 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:52.989335 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:52.989488 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:52.989622 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:52.992371 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:52.994550 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:52.994596 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:52.994774 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:52.997625 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:52.998148 [ 32284 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:52.998868 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:53.001374 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:53.001427 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:53.001504 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:53.001594 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:53.005400 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:53.005647 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:53.006318 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:53.006923 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:53.008157 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:53.008424 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:53.010996 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:53.011102 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:53.011207 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:53.011399 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.011556 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.011686 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:53.011755 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:53.011890 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:53.012505 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:53.012604 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:53.012699 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:53.012860 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.012986 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.013096 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:53.013157 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:53.013275 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:53.014003 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:53.014088 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:53.014183 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:53.014339 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.014461 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.014575 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:53.014639 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:53.014753 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:53.015357 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:53.015435 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:53.015531 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:53.015690 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.015812 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.015901 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:53.015956 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:53.016061 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:53.016702 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:53.016781 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:53.016869 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:53.017019 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.017140 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.017231 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:53.017289 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:53.017403 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:53.018002 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:53.018079 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:53.018165 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:53.018326 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.018447 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.018561 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:53.018620 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:53.018739 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:53.019221 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:53.019300 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:53.019391 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:53.019561 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.019709 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.019806 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:53.019864 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:53.019986 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:53.020354 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:53.020813 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:53.021075 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:53.021137 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:53.024823 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:53.025455 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:53.025828 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:53.026241 [ 31832 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:53.026511 [ 31832 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:53.211883 [ 31827 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1397ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:53.217486 [ 31831 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 3587ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:53.287075 [ 31841 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:53.291152 [ 31841 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:53.300807 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:53.301873 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:53.324861 [ 32474 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:53.325012 [ 32474 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:53.325314 [ 32474 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006704878 sec. (149.145 rows/sec., 3.50 KiB/sec.)
2026.05.06 08:49:53.325403 [ 32474 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:53.331955 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:53.332782 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:53.341401 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:53.342161 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:53.354982 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:53.358788 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:53.359240 [ 32275 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:53.485927 [ 31806 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1030ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:53.690127 [ 31782 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:53.690599 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:53.690645 [ 31782 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:53.690971 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:53.691080 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:53.691554 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:53.693589 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:53.693663 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:53.695453 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:53.695655 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:53.695699 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:53.695787 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:53.695856 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:53.696040 [ 31785 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:53.696362 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:53.696423 [ 31785 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:53.696494 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:53.696573 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:53.696648 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:53.696706 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:53.696776 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:53.697044 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:53.697049 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:53.698392 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:53.698558 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:53.698823 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:53.698972 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:53.699086 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:53.699393 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:53.699962 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:53.701226 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:53.701318 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:53.701386 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:53.701385 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:53.701440 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:53.701976 [ 31789 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:53.702370 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:53.702409 [ 31789 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:53.702700 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:53.702712 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:53.703044 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:53.704113 [ 31790 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:53.704174 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:53.704269 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:53.704687 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:53.704905 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:53.706115 [ 31790 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 759ms, ratio: 0.025641028, points per minute: 20.269587)
2026.05.06 08:49:53.706227 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:53.706284 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:53.706280 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:53.706658 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:53.706852 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:53.706908 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:53.706963 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:53.707031 [ 31780 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:53.707248 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:53.707275 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:53.707354 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:53.707414 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:53.707557 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:53.707666 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:53.708741 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:53.709056 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:53.709293 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:53.709393 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:53.709655 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:53.709725 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:53.710073 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:53.710156 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:53.710191 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:53.711343 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:53.711425 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:53.711485 [ 31790 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:53.711997 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:53.712010 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:53.712497 [ 31786 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:53.713063 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:53.713337 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:53.714330 [ 31790 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:53.714391 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:53.714467 [ 31790 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:53.714763 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:53.714837 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:53.714958 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:53.715075 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:53.715643 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:53.716539 [ 31790 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 10ms, ratio: 0.025641028, points per minute: 1538.4617)
2026.05.06 08:49:53.717474 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:53.718366 [ 31790 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:53.718581 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:53.718734 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:53.718843 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:53.719916 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:53.720104 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:53.720556 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:53.720565 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:53.721143 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:53.721866 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:53.721961 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:53.722145 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:53.722533 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:53.724192 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:53.726986 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:53.727068 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:53.727169 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:53.727195 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:53.727314 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:53.727317 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:53.727508 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:53.727846 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.728066 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.728226 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:53.728324 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:53.728546 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:53.729490 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:53.729637 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:53.729779 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:53.730035 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.730265 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.730466 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:53.730593 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:53.730814 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:53.732022 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:53.732156 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:53.732164 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:53.732297 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:53.732352 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:53.732571 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.732773 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.732945 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:53.732946 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:53.733044 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:53.733234 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:53.733541 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:53.734310 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:53.734442 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:53.734559 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:53.734605 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:53.734862 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.734954 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:53.735062 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.735220 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:53.735316 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:53.735508 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:53.736656 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:53.736802 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:53.736948 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:53.737205 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.737416 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.737588 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:53.737685 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:53.737883 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:53.739058 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:53.739193 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:53.739344 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:53.739542 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:53.739672 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.739733 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:53.739888 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:53.739903 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.740064 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:53.740173 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:53.740176 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.740396 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.740396 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:53.740590 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:53.740690 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:53.740899 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:53.741294 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:53.741425 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:53.741585 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:53.741843 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.741853 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:53.741990 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:53.742047 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:53.742150 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:53.742214 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:53.742308 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:53.742409 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.742501 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:53.742625 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.742783 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:53.742876 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:53.743061 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:53.743069 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:53.743854 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:53.744261 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:53.744358 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:53.744396 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:53.744455 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:53.744559 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:53.744838 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.745040 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.745215 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:53.745310 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:53.745504 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:53.746487 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:53.746634 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:53.746774 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:53.747033 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.747243 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.747397 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:53.747491 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:53.747730 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:53.747746 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:53.748847 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:53.748976 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:53.749119 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:53.749078 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:53.749375 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.749594 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.749750 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:53.749736 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:53.749853 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:53.750052 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:53.751027 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:53.751153 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:53.751295 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:53.751567 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.751799 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.751953 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:53.752042 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:53.752234 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:53.752988 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:53.753108 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:53.753241 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:53.753500 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.753715 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:53.753860 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:53.753951 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:53.754140 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:53.754676 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:53.755343 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:53.755861 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:53.755958 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:53.759927 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:53.761406 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:53.762130 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:53.762457 [ 31773 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:53.870850 [ 31769 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:53.876820 [ 31769 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:53.885932 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:53.886973 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:53.908890 [ 32349 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:53.909038 [ 32349 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:53.909331 [ 32349 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006366338 sec. (157.076 rows/sec., 3.68 KiB/sec.)
2026.05.06 08:49:53.909418 [ 32349 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:53.914584 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:53.915439 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:53.923195 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:53.924027 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:53.937648 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:53.941542 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:53.942119 [ 32280 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:53.962993 [ 32317 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:49:53.963500 [ 32317 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:49:53 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE73B1BEF7C4C
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:49:53.963965 [ 32317 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:49:54.334058 [ 31762 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:54.334588 [ 31762 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:54.334646 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:54.334859 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:54.335098 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:54.335137 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:54.337247 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:54.337262 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:54.338822 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:54.338925 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:54.339019 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:54.339102 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:54.339166 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:54.339337 [ 31757 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:54.339748 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:54.339769 [ 31757 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:54.339775 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:54.339908 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:54.339980 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:54.340054 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:54.340066 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:54.340319 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:54.340374 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:54.341430 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:54.341869 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:54.341866 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:54.341943 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:54.342081 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:54.342348 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:54.342816 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:54.343799 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:54.343890 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:54.343956 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:54.344007 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:54.344078 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:54.344372 [ 31748 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:54.344838 [ 31748 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:54.344835 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:54.345162 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:54.345288 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:54.345717 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:54.346734 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:54.346747 [ 31759 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:54.346848 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:54.346944 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:54.347268 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:54.348225 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:54.348627 [ 31770 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:54.348807 [ 31759 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 631ms, ratio: 0.025641028, points per minute: 24.381325)
2026.05.06 08:49:54.348845 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:54.348930 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:54.348941 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:54.348993 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:54.349096 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:54.349489 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:54.349559 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:54.349619 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:54.349672 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:54.349796 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:54.349905 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:54.349916 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:54.350440 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:54.351060 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:54.351635 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:54.351742 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:54.351904 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:54.352014 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:54.352200 [ 31751 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:54.352433 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:54.352563 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:54.352766 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:54.352823 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:54.353699 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:54.353781 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:54.353843 [ 31759 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:54.354744 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:54.355150 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:54.355302 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:54.356701 [ 31759 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:54.356771 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:54.356851 [ 31759 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:54.356846 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:54.357749 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:54.357873 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:54.358018 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:54.358793 [ 31759 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 9ms, ratio: 0.025641028, points per minute: 1709.4017)
2026.05.06 08:49:54.360193 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:54.360816 [ 31759 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:54.362509 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:54.363113 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:54.363195 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:54.363323 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:54.363441 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:54.364971 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:54.365170 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:54.369030 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:54.369280 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:54.369907 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:54.370478 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:54.371583 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:54.371806 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:54.371910 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:54.371987 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:54.372102 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:54.372127 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:54.372319 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:54.377089 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:54.377294 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:54.377816 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:54.377950 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:54.378165 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:54.378266 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:54.378347 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:54.378693 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.378934 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.379174 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:54.379193 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:54.379315 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:54.379561 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:54.379681 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:54.380708 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:54.380860 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:54.381056 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:54.381381 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.381645 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.381820 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:54.381925 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:54.382190 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:54.383086 [ 31754 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 10244ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:54.383488 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:54.383708 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:54.383873 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:54.384208 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:54.384213 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.384375 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:54.384543 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.384565 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:54.384725 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:54.384835 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:54.384860 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.385101 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.385086 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:54.385277 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:54.385379 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:54.385608 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:54.386354 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:54.386510 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:54.386569 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:54.386691 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:54.386706 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:54.386855 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:54.386999 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.387126 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.387271 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.387339 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.387475 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:54.387501 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:54.387641 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:54.387650 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:54.387872 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:54.387889 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:54.388990 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:54.389124 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:54.389221 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:54.389271 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:54.389378 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:54.389553 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.389563 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:54.389776 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.389902 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.389957 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:54.390057 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:54.390188 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.390258 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:54.390382 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:54.390494 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:54.390741 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:54.391261 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:54.391392 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:54.391555 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:54.391855 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.391993 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:54.392064 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.392182 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:54.392241 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:54.392338 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:54.392358 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:54.392552 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:54.392677 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.392906 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.393131 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:54.393244 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:54.393464 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:54.393635 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:54.393809 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:54.393960 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:54.394225 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.394413 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:54.394453 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.394567 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:54.394635 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:54.394712 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:54.394750 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:54.394957 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:54.394987 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.395235 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:54.395422 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:54.395571 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:54.395994 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:54.395982 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:54.396141 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:54.396290 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:54.396590 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.396696 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:54.396795 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.396954 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:54.397059 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:54.397279 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:54.397477 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:54.398201 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:54.398238 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:54.398330 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:54.398343 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:54.398478 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:54.398764 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.398970 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:54.399119 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:54.399227 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:54.399425 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:54.400015 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:54.400706 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:54.401110 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:54.401201 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:54.403098 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:54.404491 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:54.404882 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:54.405222 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:54.406181 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:54.406828 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:54.407249 [ 31746 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:54.407462 [ 31746 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:54.680976 [ 31740 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1195ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:54.915570 [ 31736 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:54.921681 [ 31736 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:54.933097 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:54.934209 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:54.959463 [ 32429 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:54.959716 [ 32429 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:54.960033 [ 32429 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007716369 sec. (129.595 rows/sec., 3.04 KiB/sec.)
2026.05.06 08:49:54.960143 [ 32429 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:54.964545 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:54.965202 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:54.972355 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:54.973068 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:54.986041 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:54.989881 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:54.990478 [ 32284 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:55.096608 [ 31640 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #5
2026.05.06 08:49:55.096665 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:49:55.097052 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:55.102953 [ 31734 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.103359 [ 31734 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.103494 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:55.103719 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.104038 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.104062 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.106795 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.106877 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.108868 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.108903 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.109145 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:55.109426 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:55.109567 [ 31742 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.109612 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:55.109897 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:55.109985 [ 31742 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.110198 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.110282 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.110605 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:55.110676 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.110717 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:55.110836 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:55.111430 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:55.112107 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.112798 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.113297 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:55.113900 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.114103 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:55.114214 [ 31727 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.114334 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:55.114568 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:55.114623 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.114762 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:55.114867 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.115234 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:55.116471 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:55.116621 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:55.116732 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:55.117217 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.119080 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.119467 [ 31725 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.119917 [ 31725 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.119920 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.120223 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:55.120251 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.120506 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.122019 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.122106 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.122196 [ 31728 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:55.122294 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:55.122460 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:55.123662 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.123757 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:55.124137 [ 31720 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.124490 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:55.124576 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.124610 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:55.124711 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:55.124897 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.124916 [ 31728 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 764ms, ratio: 0.025641028, points per minute: 20.136932)
2026.05.06 08:49:55.125175 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:55.125318 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:55.126040 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:55.126148 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:55.126248 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:55.126483 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:55.126764 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:55.126857 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.127866 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:55.127973 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:55.128054 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:55.128469 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.128797 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:55.128989 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:55.129326 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:55.129747 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:55.130315 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:55.131004 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:55.131151 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:55.131269 [ 31728 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:55.132896 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:55.132973 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:55.133091 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:55.133204 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:55.136435 [ 31728 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:55.136554 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:55.136673 [ 31728 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:55.138006 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:55.138184 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:55.138579 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:55.139050 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:55.140017 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:55.140153 [ 31728 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:55.140380 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:55.143060 [ 31728 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:55.144909 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:55.145055 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:55.145220 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:55.145498 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.145727 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.145886 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:55.145982 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:55.146188 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:55.147112 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:55.147238 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:55.147381 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:55.147676 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.147880 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.148033 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:55.148126 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:55.148317 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:55.149386 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:55.149511 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:55.149668 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:55.149935 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.150132 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.150288 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:55.150383 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:55.150588 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:55.151557 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:55.151709 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:55.151851 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:55.152101 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.152295 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.152447 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:55.152553 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:55.152754 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:55.153817 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:55.153943 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:55.154085 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:55.154340 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.154560 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.154715 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:55.154810 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:55.155004 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:55.155997 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:55.156123 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:55.156264 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:55.156509 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.156718 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.156879 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:55.156967 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:55.157158 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:55.157906 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:55.158024 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:55.158155 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:55.158398 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.158612 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:55.158756 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:55.158846 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:55.159035 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:55.159567 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:55.160258 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:55.160679 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:55.160770 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:55.164416 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:55.165942 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:55.166644 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:55.167067 [ 31712 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.167445 [ 31712 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:55.167482 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.167950 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.169903 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.171861 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:55.172811 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:55.172928 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:55.173036 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:55.175941 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:55.178227 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:55.178373 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:55.182366 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:55.184253 [ 31714 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:55.185157 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:55.185269 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:55.185434 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:55.185616 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:55.190415 [ 31714 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:55.191250 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:55.191448 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:55.191986 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:55.192685 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:55.193692 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:55.194096 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:55.198903 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:55.199063 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:55.199233 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:55.199541 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.199749 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:55.199795 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.199974 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:55.200074 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:55.200282 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:55.200793 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:55.201312 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:55.201454 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:55.201622 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:55.201896 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.202109 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.202269 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:55.202365 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:55.202586 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:55.203821 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:55.203958 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:55.204102 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:55.204362 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.204580 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.204740 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:55.204839 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:55.205037 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:55.206049 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:55.206178 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:55.206322 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:55.206592 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.206833 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.207008 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:55.207121 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:55.207321 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:55.208482 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:55.208632 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:55.208780 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:55.209039 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.209248 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.209408 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:55.209505 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:55.209724 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:55.210724 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:55.210853 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:55.211001 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:55.211254 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.211456 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.211687 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:55.211784 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:55.211983 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:55.212764 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:55.212888 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:55.213027 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:55.213278 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.213478 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:55.213641 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:55.213733 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:55.213929 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:55.214481 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:55.215293 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:55.215772 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:55.215885 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:55.219459 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:55.220849 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:55.221499 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:55.221921 [ 31750 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:55.223145 [ 32400 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:55.223285 [ 32400 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:55.223597 [ 32400 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006550329 sec. (152.664 rows/sec., 3.58 KiB/sec.)
2026.05.06 08:49:55.223736 [ 32400 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:55.227675 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:55.228345 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:55.235317 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:55.235948 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:55.247351 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:55.250315 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:55.250798 [ 32275 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:55.401682 [ 31705 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.402176 [ 31705 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.402187 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:55.402556 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.402620 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.402883 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.405157 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.405206 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.407043 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.407079 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.407337 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:55.407472 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:55.407595 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:55.407688 [ 31699 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.408068 [ 31699 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.408153 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:55.408341 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:55.408356 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.408485 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:55.408585 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.408618 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:55.408702 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.409216 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:55.410286 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.410396 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.411144 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:55.412098 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:55.412307 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:55.412579 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.412691 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:55.412771 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.412994 [ 31698 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.413202 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:55.413390 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:55.413406 [ 31698 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.413712 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.413792 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.414177 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.414604 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:55.414752 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:55.414885 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:55.415506 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.415934 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.417140 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.417576 [ 31695 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.417604 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.417998 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:55.418070 [ 31695 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.418336 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.418337 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.418645 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.420091 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.420236 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.420625 [ 31706 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:55.420755 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:55.420883 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:55.421757 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:55.421819 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.422165 [ 31688 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.422447 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.422502 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:55.422596 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.422627 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:55.422727 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:55.423397 [ 31706 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 280ms, ratio: 0.025641028, points per minute: 54.945057)
2026.05.06 08:49:55.423601 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:55.423779 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:55.424115 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.424193 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:55.424366 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:55.424452 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:55.424560 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:55.425082 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:55.425483 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:55.425495 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.425675 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:55.425926 [ 31687 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.426259 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.426508 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.426603 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:55.427474 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:55.427754 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:55.428116 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:55.428542 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:55.429527 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.429614 [ 31689 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9824ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:55.429692 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:55.429985 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:55.430123 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:55.430248 [ 31706 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:55.431534 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:55.432324 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:55.432444 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:55.432632 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:55.432793 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:55.436920 [ 31706 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:55.437021 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:55.437132 [ 31706 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:55.437861 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:55.438036 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:55.438476 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:55.439016 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:55.440108 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:55.440455 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:55.440915 [ 31706 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 18ms, ratio: 0.025641028, points per minute: 854.70087)
2026.05.06 08:49:55.443161 [ 31706 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:55.444935 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:55.445076 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:55.445225 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:55.445501 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.445731 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.445889 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:55.445984 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:55.446187 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:55.447096 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:55.447221 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:55.447361 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:55.447661 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.447864 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.448016 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:55.448121 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:55.448310 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:55.449360 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:55.449483 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:55.449637 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:55.449883 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.450076 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.450229 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:55.450321 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:55.450506 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:55.451463 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:55.451600 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:55.451779 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:55.452024 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.452215 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.452378 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:55.452469 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:55.452669 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:55.453717 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:55.453841 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:55.453982 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:55.454232 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.454450 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.454618 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:55.454713 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:55.454906 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:55.455890 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:55.456014 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:55.456153 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:55.456401 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.456627 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.456777 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:55.456866 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:55.457056 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:55.457792 [ 32276 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:55.457909 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:55.458042 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:55.458283 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.458477 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32276, kind: Regular, unbound: 0
2026.05.06 08:49:55.458639 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:55.458729 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:55.458920 [ 32276 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:55.459425 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:55.460131 [ 32276 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:55.460552 [ 32276 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:55.460641 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:55.464022 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:55.465313 [ 32276 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:55.465951 [ 32276 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:55.466328 [ 31898 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:55.466606 [ 31898 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:55.466683 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:55.467013 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:55.469056 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:55.470812 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:55.471673 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:55.471793 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:55.471894 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:55.473763 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:55.475158 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:55.475266 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:55.477741 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:55.480297 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:55.480378 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:55.480500 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:55.480629 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:55.485564 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:55.485755 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:55.486261 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:55.486850 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:55.487934 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:55.488337 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:55.493053 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:55.493219 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:55.493378 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:55.493704 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.493921 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.494082 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:55.494202 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:55.494408 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:55.495359 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:55.495492 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:55.495692 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:55.495962 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.496168 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.496324 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:55.496420 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:55.496650 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:55.497809 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:55.497976 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:55.498143 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:55.498412 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.498659 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.498826 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:55.498924 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:55.499157 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:55.500260 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:55.500423 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:55.500611 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:55.500885 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.501090 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.501249 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:55.501362 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:55.501576 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:55.502680 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:55.502811 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:55.502957 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:55.503232 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.503441 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.503673 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:55.503775 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:55.503976 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:55.505017 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:55.505147 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:55.505291 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:55.505569 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.505768 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.505955 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:55.506058 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:55.506254 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:55.507033 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:55.507157 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:55.507295 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:55.507565 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.507800 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:55.507953 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:55.508045 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:55.508239 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:55.508795 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:55.509458 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:55.509875 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:55.509967 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:55.514551 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:55.515902 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:55.516551 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:55.516971 [ 31696 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:55.517152 [ 31696 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:55.734427 [ 31697 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1351ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:55.739642 [ 31694 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 4303ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:55.903117 [ 32154 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 8843ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:56.009784 [ 32142 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.010212 [ 32142 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.010309 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:56.010562 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.010732 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.010912 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.013255 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.013279 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.015132 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.015149 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.015379 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:56.015493 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:56.015599 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:56.015709 [ 32143 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.016031 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:56.016101 [ 32143 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.016303 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.016361 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.016399 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:56.016503 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:56.016631 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:56.016696 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.017118 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:56.018680 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.018764 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:56.018838 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.019634 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:56.019821 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:56.020124 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:56.020642 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.021614 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.021695 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.021951 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:56.022062 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:56.022153 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:56.022198 [ 32069 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.022701 [ 32069 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.022725 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:56.022982 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.023137 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.023256 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.025347 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.025409 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.026319 [ 32068 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:56.026404 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:56.026543 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:56.027379 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:56.027435 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.027874 [ 32075 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.028196 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.028297 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:56.028335 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.028412 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:56.028534 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:56.028841 [ 32068 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 586ms, ratio: 0.025641028, points per minute: 26.253613)
2026.05.06 08:49:56.028987 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:56.029060 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:56.029606 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:56.029675 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:56.029748 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:56.029779 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.030122 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:56.031311 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.031317 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:56.031483 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:56.031759 [ 31873 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.032185 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.032266 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:56.032407 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:56.032537 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.032819 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:56.033278 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.033654 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:56.033800 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:56.034543 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:56.034652 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:56.034738 [ 32068 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:56.035150 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.036813 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.037186 [ 32063 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.037494 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.037531 [ 32079 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1356ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:56.037797 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.037846 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:56.038884 [ 32068 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:56.038962 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:56.039056 [ 32068 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:56.039868 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.040017 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:56.040092 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:56.040211 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:56.040323 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:56.041036 [ 32068 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 12ms, ratio: 0.025641028, points per minute: 1282.0513)
2026.05.06 08:49:56.041549 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:56.042359 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:56.042470 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:56.042589 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:56.042820 [ 32068 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:56.045078 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:56.045263 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:56.045334 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:56.045656 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:56.046138 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.047105 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:56.047470 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:56.047596 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:56.047759 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:56.051650 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:56.052110 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:56.052276 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:56.052454 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:56.052770 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.053016 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.053184 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:56.053283 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:56.053509 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:56.054032 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:56.054113 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:56.054235 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:56.054351 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:56.054546 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:56.054687 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:56.054841 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:56.055153 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.055395 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.055632 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:56.055742 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:56.055946 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:56.057228 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:56.057373 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:56.057549 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.057824 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.058033 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.058213 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.058344 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.058623 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.059195 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:56.059379 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:56.059931 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:56.059944 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:56.060087 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:56.060236 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:56.060446 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.060506 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.060732 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.060901 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:56.060996 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:56.061194 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:56.061382 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:56.061790 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:56.062277 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:56.062425 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:56.062595 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:56.062861 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.063076 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.063237 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:56.063343 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:56.063560 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:56.064670 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:56.064811 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:56.064963 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.065235 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.065459 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.065635 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.065729 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.065933 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.066420 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:56.066585 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:56.066734 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:56.066738 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:56.066878 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:56.067023 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:56.067092 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.067284 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.067311 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.067474 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:56.067497 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:49:56.067589 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:56.067701 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:56.067797 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:56.067851 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:56.067999 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:56.068575 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:56.068987 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:56.069152 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:56.069270 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:56.069301 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:56.069585 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.069700 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:56.069795 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:56.069790 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.069985 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:56.070081 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:56.070287 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:56.071362 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:56.071492 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:56.071686 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.071971 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.072173 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.072331 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.072437 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.072661 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.073628 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:56.073658 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:56.073787 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:56.073928 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:56.074177 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.074373 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.074543 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:56.074637 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:56.074837 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:56.074972 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.075688 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.075967 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:56.076099 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:56.076243 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:56.076496 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.076753 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.076914 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:56.077010 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:56.077215 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:56.078182 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:56.078307 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:56.078449 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.078715 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.078912 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.079063 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.079152 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.079342 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.080210 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:56.080345 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:56.080488 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:56.080820 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.081047 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.081198 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:56.081291 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:56.081483 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:56.082056 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:56.082734 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:56.083140 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:56.083231 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:56.087002 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:56.088552 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.089304 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.089739 [ 32107 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:56.089923 [ 32107 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:56.100941 [ 31874 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:56.107240 [ 31874 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:56.116506 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:56.117617 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:56.142507 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.143023 [ 32272 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.212572 [ 31887 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.212984 [ 31887 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:56.213094 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:56.213460 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.216303 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.218267 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.218530 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:56.218682 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:56.218787 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:56.218811 [ 32073 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.219195 [ 32073 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.219191 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.219495 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:56.219513 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.219760 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:56.219904 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.219942 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:56.220110 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:56.220824 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:56.221323 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.221617 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.222415 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:56.222955 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.223149 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:56.223373 [ 32061 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.223363 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:56.223382 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.223851 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:56.223896 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.223950 [ 32061 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.224278 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.224286 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:56.224419 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.224634 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.225740 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:56.225899 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:56.226060 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:56.226314 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.226440 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.227968 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.228101 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.228333 [ 31867 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.229210 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.229233 [ 31867 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.229478 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.229552 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:56.229832 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.231309 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.231570 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.231911 [ 31885 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:56.232055 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:56.232214 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:56.233095 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.233541 [ 32064 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.233583 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:56.234011 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.234368 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:56.234418 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.234478 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:56.234598 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:56.235067 [ 31885 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 192ms, ratio: 0.025641028, points per minute: 80.128204)
2026.05.06 08:49:56.235302 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:56.235420 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:56.236150 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:56.236275 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:56.236392 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:56.236513 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.237017 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:56.237648 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:56.238365 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.238497 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:56.238803 [ 32185 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.239146 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.239247 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:56.239454 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:56.239474 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.239856 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:56.240193 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:56.240258 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.240341 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:56.241556 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:56.241699 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:56.241810 [ 31885 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:56.242199 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.244112 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.244826 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:56.246725 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:56.246800 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:56.246927 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:56.247040 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:56.247813 [ 31885 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:56.247950 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:56.248114 [ 31885 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:56.250704 [ 31885 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:56.251851 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:56.252035 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:56.252448 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:56.253061 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.253954 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:56.254103 [ 31885 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 4ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:56.254377 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:56.258965 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:56.259108 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:56.259257 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:56.259551 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.259799 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.259969 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:56.260065 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:56.260268 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:56.261201 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:56.261326 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:56.261466 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:56.261759 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.261958 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.262113 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:56.262204 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:56.262396 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:56.263443 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:56.263581 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:56.263757 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.264010 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.264208 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.264362 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.264456 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.264658 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.265618 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:56.265760 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:56.265898 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:56.266142 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.266335 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.266486 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:56.266596 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:56.266783 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:56.267870 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:56.268007 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:56.268149 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:56.268397 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.268614 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.268767 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:56.268861 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:56.269065 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:56.270091 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:56.270216 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:56.270370 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.270634 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.270829 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.270977 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.271065 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.271254 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.272051 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:56.272173 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:56.272307 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:56.272565 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.272761 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:56.272905 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:56.273017 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:56.273206 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:56.273741 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:56.274385 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:56.274800 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:56.274891 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:56.278283 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:56.279809 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.280544 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.280927 [ 31925 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:56.281168 [ 31925 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.281528 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.281558 [ 31925 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:56.281863 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.283796 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.285635 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:56.286458 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:56.286595 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:56.286699 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:56.289486 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:56.291867 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:56.292026 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:56.296133 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:56.298491 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:56.298587 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:56.298715 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:56.298849 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:56.303870 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:56.304063 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:56.304590 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:56.305138 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.306246 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:56.306673 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:56.311226 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:56.311377 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:56.311550 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:56.311877 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.312135 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.312303 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:56.312402 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:56.312666 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:56.313626 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:56.313759 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:56.313904 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:56.314188 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.314405 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.314582 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:56.314676 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:56.314868 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:56.316001 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:56.316135 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:56.316274 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.316546 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.316749 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.316905 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.317000 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.317190 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.318176 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:56.318316 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:56.318455 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:56.318725 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.318926 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.319079 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:56.319173 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:56.319359 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:56.320482 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:56.320633 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:56.320775 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:56.321031 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.321235 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.321389 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:56.321483 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:56.321693 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:56.322689 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:56.322816 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:56.322958 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.323210 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.323407 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.323573 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.323707 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.323903 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.324710 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:56.324870 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:56.325010 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:56.325267 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.325463 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:49:56.325630 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:56.325744 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:56.325938 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:56.326461 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:56.327138 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:56.327560 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:56.327711 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:56.331250 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:56.332799 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.333493 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.333987 [ 32127 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:56.334186 [ 32127 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:56.655982 [ 32092 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:56.662050 [ 32092 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:56.671108 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:56.672186 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:56.697767 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.698295 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0917
8. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003afe2f18
9. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
10. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
11. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
12. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
13. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
14. ? @ 0x00007f6e2fbe4ac3
15. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.781834 [ 32158 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.782269 [ 32158 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.782328 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:56.782603 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.782810 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.782896 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.785728 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.785760 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.788003 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.788101 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.788327 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:56.788575 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:56.788710 [ 32141 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.788725 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:56.789154 [ 32141 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.789224 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:56.789469 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.789643 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.789661 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:56.789824 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.789867 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:56.789989 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:56.790713 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:56.791923 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.791946 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.792589 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:56.793342 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:56.793673 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:56.794142 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:56.794166 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.794230 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.794659 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.794711 [ 32126 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.795004 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:56.795068 [ 32126 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.795330 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.795416 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.795777 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.796266 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:56.796417 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:56.796594 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:56.797359 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.797508 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.799443 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.799497 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.799871 [ 32094 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.800232 [ 32094 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.800252 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:56.800556 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.800636 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.800837 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.802487 [ 32148 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:56.802641 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.802654 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.802650 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:56.802954 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:56.804730 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:56.804895 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.805293 [ 32168 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.805643 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.805722 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:56.805812 [ 32148 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 551ms, ratio: 0.025641028, points per minute: 27.921263)
2026.05.06 08:49:56.805843 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:56.805921 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.805950 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:56.806005 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:56.806104 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:56.806787 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:56.806893 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:56.806993 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:56.807512 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:56.808043 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.808760 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:56.809229 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:56.809843 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.809973 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:56.810162 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:56.810275 [ 32085 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.810676 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:56.810887 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.811180 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.811187 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:56.811266 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.811396 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:56.813235 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:56.813377 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:56.813485 [ 32148 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:56.814341 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.815804 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:56.817071 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:56.817159 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:56.817283 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:56.817401 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:56.818000 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:56.819658 [ 32148 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:56.819764 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:56.819880 [ 32148 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:56.822417 [ 32148 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:49:56.822421 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:56.822683 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:56.823119 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:56.823750 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.824641 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:56.825028 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:56.825652 [ 32148 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:56.829861 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:56.830005 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:56.830156 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:56.830432 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.830661 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.830821 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:56.830916 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:56.831120 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:56.832117 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:56.832249 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:56.832389 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:56.832660 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.832881 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.833035 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:56.833128 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:56.833318 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:56.834399 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:56.834539 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:56.834685 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.834932 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.835126 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.835279 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.835372 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.835574 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.836607 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:56.836741 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:56.836878 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:56.837138 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.837328 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.837477 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:56.837584 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:56.837769 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:56.838857 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:56.838981 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:56.839119 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:56.839368 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.839581 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.839790 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:56.839885 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:56.840080 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:56.841072 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:56.841206 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:56.841345 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.841610 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.841802 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.841949 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.842036 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.842225 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.842989 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:56.843105 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:56.843237 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:56.843479 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.843718 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:56.843865 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:56.843956 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:56.844145 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:56.844691 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:56.845338 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:56.845745 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:56.845834 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:56.849420 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:56.850935 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.851666 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.852084 [ 32173 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:56.852317 [ 32173 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:56.852650 [ 32173 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:56.852668 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:56.852940 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:56.854648 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:56.856260 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:56.857042 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:56.857140 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:56.857235 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:56.860008 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:56.862319 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:56.862473 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:56.866536 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:56.868657 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:56.868734 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:56.868843 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:56.868948 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:56.873289 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:56.873477 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:56.874014 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:56.874712 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:56.875855 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:56.876213 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:56.880652 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:56.880786 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:56.880922 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:56.881173 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.881368 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.881513 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:56.881648 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:56.881835 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:56.882698 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:56.882817 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:56.882948 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:56.883187 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.883388 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.883552 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:56.883672 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:56.883855 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:56.884843 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:56.884958 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:56.885087 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.885322 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.885502 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.885667 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.885755 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.885932 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.886834 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:56.886949 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:56.887077 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:56.887324 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.887506 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.887719 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:56.887807 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:56.887982 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:56.888973 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:56.889088 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:56.889245 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:56.889584 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.889826 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.890009 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:56.890124 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:56.890298 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:56.891121 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:56.891238 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:56.891356 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:56.891575 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.891757 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.891880 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:56.891948 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:56.892095 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:56.892680 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:56.892773 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:56.892877 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:56.893069 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.893221 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:56.893344 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:56.893414 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:56.893577 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:56.893977 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:56.894474 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:56.894793 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:56.894861 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:56.896609 [ 32120 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:56.899294 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:56.900368 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.900894 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.901278 [ 32180 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:56.901469 [ 32180 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:56.902053 [ 32120 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:56.909582 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:56.910572 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:56.932272 [ 32390 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:49:56.932428 [ 32390 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:49:56.932769 [ 32390 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006778269 sec. (147.530 rows/sec., 3.46 KiB/sec.)
2026.05.06 08:49:56.932859 [ 32390 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:49:56.937339 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:49:56.938063 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:56.945004 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:49:56.945809 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:49:56.957252 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.960532 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:56.961027 [ 32274 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:49:57.526257 [ 32153 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1489ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:57.608141 [ 32144 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:57.608697 [ 32144 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:57.608724 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:57.609089 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:57.609205 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:57.609371 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:57.611968 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:57.612041 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:57.613803 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:57.613871 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:57.614174 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:57.614398 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:57.614539 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:57.614545 [ 32163 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:57.614984 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:57.615021 [ 32163 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:57.615400 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:57.615445 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:57.615445 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:57.615576 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:57.615712 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:57.615748 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:57.616376 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:57.617800 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:57.617856 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:57.617975 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:57.618772 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:57.618958 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:57.619408 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:57.619923 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:57.620981 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:57.621051 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:57.621469 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:57.621452 [ 32136 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:57.621642 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:57.621757 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:57.621884 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:57.621952 [ 32136 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:57.622245 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:57.622307 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:57.622595 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:57.624121 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:57.624420 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:57.625940 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:57.626238 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:57.626411 [ 32077 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:57.626785 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:57.626837 [ 32077 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:57.626994 [ 32184 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:57.627118 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:57.627134 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:57.627203 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:57.627276 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:57.627643 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:57.629042 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:57.629566 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:57.629675 [ 32184 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 804ms, ratio: 0.025641028, points per minute: 19.135096)
2026.05.06 08:49:57.629861 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:57.629951 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:57.630503 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:57.630609 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:57.630697 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:57.630873 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:57.631146 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:57.631562 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:57.631674 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:57.631788 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:57.631891 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:57.632038 [ 32139 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:57.632439 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:57.632649 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:57.632735 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:57.633451 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:57.633673 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:57.634016 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:57.634432 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:57.634632 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:57.635324 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:57.635754 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:57.635896 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:57.636009 [ 32184 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:57.637114 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:57.637147 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:57.637310 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:57.637491 [ 32123 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:57.637825 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:57.638071 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:57.639852 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:57.641485 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:57.641867 [ 32192 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:57.642176 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:57.642438 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:57.642766 [ 32184 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:57.642916 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:57.643078 [ 32184 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:57.644198 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:57.644412 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:57.646058 [ 32184 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:49:57.646138 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:57.646632 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:57.646707 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:57.646827 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:57.646941 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:57.646965 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:57.647064 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:57.647155 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:57.648224 [ 32184 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:57.650393 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:57.651676 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:57.651866 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:57.652308 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:57.652919 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:57.653277 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:57.653440 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:57.653896 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:57.654287 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:57.657791 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:57.658949 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:57.659103 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:57.659269 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:57.659576 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.659855 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.659984 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:57.660020 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:57.660054 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:57.660118 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:57.660162 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:57.660264 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:57.660326 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:57.661281 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:57.661412 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:57.661573 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:57.661836 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.662052 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.662209 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:57.662304 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:57.662496 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:57.663595 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:57.663763 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:57.663907 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:57.664166 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.664363 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.664540 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:57.664607 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:57.664636 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:57.664775 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:57.664828 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:57.665232 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:57.665724 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:57.665829 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:57.665956 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:57.666098 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:57.666352 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.666566 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.666639 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:57.666721 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:57.666816 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:57.666979 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:57.667005 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:57.668127 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:57.668259 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:57.668403 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:57.668675 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.668881 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.669050 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:57.669144 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:57.669341 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:57.670328 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:57.670454 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:57.670614 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:57.670867 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.671067 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.671219 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:57.671308 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:57.671328 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:57.671456 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:57.671501 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:57.671635 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:57.671883 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.672086 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.672232 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:57.672293 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:57.672316 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:57.672414 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:57.672494 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:57.672565 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:57.672814 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.673011 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:49:57.673157 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:57.673246 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:57.673340 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:57.673434 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:57.673454 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:57.673599 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:57.673847 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.673970 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:57.674032 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.674169 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:57.674252 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:57.674461 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:57.674685 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:57.675082 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:57.675172 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:57.675455 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:57.675589 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:57.675749 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:57.675982 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.676160 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.676299 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:57.676411 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:57.676603 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:57.677475 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:57.677606 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:57.677728 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:57.677962 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.678141 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.678279 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:57.678363 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:57.678555 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:57.679292 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:57.679500 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:57.679671 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:57.679803 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:57.680041 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.680241 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.680380 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:57.680465 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:57.680661 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:57.680650 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:57.681294 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:57.681546 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:57.681662 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:57.681788 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:57.682020 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.682196 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.682332 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:57.682413 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:57.682599 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:57.683265 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:57.683374 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:57.683495 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:57.683773 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.683956 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:49:57.684090 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:57.684186 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:57.684363 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:57.684834 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:57.685453 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:57.685832 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:57.685913 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:57.689064 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:57.690302 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:57.690912 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:57.691303 [ 32055 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:57.774105 [ 31911 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:57.779968 [ 31911 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:57.788933 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:57.789958 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:57.814630 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:57.815156 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:58.271781 [ 32040 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 2537ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:58.357011 [ 32039 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.357462 [ 32039 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.357542 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:58.357809 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.357930 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.358098 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.360447 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.360465 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.362281 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.362359 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.362535 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:58.362752 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:58.362855 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:58.362895 [ 32031 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.363283 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:58.363362 [ 32031 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.363486 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:58.363679 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:58.363721 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.363678 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.363854 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:58.364067 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.364542 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:58.365641 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.365715 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.366032 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:58.366690 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:58.366932 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:58.367198 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.367230 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.367286 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:58.367648 [ 32034 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.367801 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:58.367964 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:58.368027 [ 32034 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.368278 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.368289 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.368613 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.368953 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:58.369109 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:58.369222 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:58.370130 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.370158 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.371878 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.371954 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.372410 [ 32041 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.372789 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:58.372806 [ 32041 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.373153 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.373155 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.373440 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.374678 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.374724 [ 32030 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:58.374822 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:58.374963 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:58.375066 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.376537 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:58.377038 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.377414 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:58.377549 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:58.377559 [ 32030 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 729ms, ratio: 0.025641028, points per minute: 21.103725)
2026.05.06 08:49:58.377651 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:58.377725 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:58.377810 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:58.378383 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:58.378464 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:58.378565 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:58.379007 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:58.380446 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:58.380462 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:58.381136 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:58.381303 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:58.381648 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:58.382037 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:58.382732 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:58.382870 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:58.383333 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:58.383446 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:58.383557 [ 32030 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:58.386803 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:58.388777 [ 32030 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:58.388869 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:58.388973 [ 32030 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:58.389084 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:58.389155 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:58.389272 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:58.389382 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:58.391285 [ 32030 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:58.393977 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:58.394156 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:58.394575 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:58.395065 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:58.396080 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:58.396430 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:58.401388 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:58.401552 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:58.401697 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:58.401971 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.402180 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.402350 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:58.402444 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:58.402663 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:58.403576 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:58.403746 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:58.403885 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:58.404134 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.404330 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.404480 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:58.404586 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:58.404774 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:58.405867 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:58.406024 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:58.406197 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:58.406463 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.406698 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.406854 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:58.406947 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:58.407135 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:58.408153 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:58.408279 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:58.408417 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:58.408700 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.408908 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.409060 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:58.409151 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:58.409337 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:58.410415 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:58.410559 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:58.410700 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:58.410966 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.411169 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.411319 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:58.411412 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:58.411649 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:58.412624 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:58.412744 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:58.412908 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:58.413160 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.413355 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.413504 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:58.413606 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:58.413795 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:58.414547 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:58.414678 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:58.414811 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:58.415053 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.415244 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:49:58.415386 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:58.415473 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:58.415728 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:58.416320 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:58.416973 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:58.417364 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:58.417454 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:58.421564 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:58.423054 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:58.423791 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:58.424206 [ 32026 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:58.424482 [ 32026 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.424820 [ 32026 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:58.424873 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.425164 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.427208 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.428977 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:58.429793 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:58.429894 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:58.430001 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:58.432874 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:58.435142 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:58.435294 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:58.439366 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:58.441651 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:58.441724 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:58.441833 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:58.441937 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:58.446389 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:58.446578 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:58.447124 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:58.447687 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:58.448798 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:58.449158 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:58.453835 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:58.453972 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:58.454110 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:58.454362 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.454576 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.454735 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:58.454825 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:58.455013 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:58.455945 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:58.456069 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:58.456200 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:58.456438 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.456659 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.456803 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:58.456888 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:58.457067 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:58.458074 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:58.458192 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:58.458322 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:58.458573 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.458783 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.458926 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:58.459013 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:58.459189 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:58.460179 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:58.460299 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:58.460427 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:58.460680 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.460861 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.461004 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:58.461090 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:58.461273 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:58.462285 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:58.462402 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:58.462553 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:58.462826 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.463017 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.463162 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:58.463251 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:58.463436 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:58.464404 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:58.464541 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:58.464672 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:58.464912 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.465096 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.465236 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:58.465320 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:58.465498 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:58.466204 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:58.466333 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:58.466459 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:58.466709 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.466896 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:49:58.467034 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:58.467118 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:58.467299 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:58.467827 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:58.468442 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:58.468831 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:58.468917 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:58.473178 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:58.474621 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:58.475291 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:58.475739 [ 32047 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:58.475908 [ 32047 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:58.802439 [ 31998 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.802851 [ 31998 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.802912 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:58.803153 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.803286 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.803439 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.806079 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.806178 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.808070 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.808180 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.808355 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:58.808544 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:58.808649 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:58.808652 [ 31999 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.809019 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.809092 [ 31999 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.809347 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:58.809404 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.809448 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:58.809460 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:58.809577 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:58.809863 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.810162 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:58.811316 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.811645 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:58.811857 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.812354 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:58.812601 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:58.813011 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:58.813134 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.813451 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:58.814438 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.814798 [ 31995 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.814840 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:58.814963 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:58.815073 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:58.815124 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.815133 [ 31995 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.815462 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.815509 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:58.815810 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.817353 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.817412 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.819050 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.819105 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.819559 [ 31993 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.819933 [ 31993 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.819948 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.820213 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:58.820237 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.820238 [ 31997 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:58.820409 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:58.820569 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:58.820625 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.823333 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.823895 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.824026 [ 31997 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 433ms, ratio: 0.025641028, points per minute: 35.53029)
2026.05.06 08:49:58.824205 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:58.824291 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:58.824872 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:58.824956 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:58.825042 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:58.825328 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.825488 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:58.825630 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:58.825672 [ 32003 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.825955 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.826198 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.826577 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:58.826699 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:58.826802 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:58.826931 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:58.827703 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:58.827873 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:58.827879 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.828203 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:58.828585 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:58.829625 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:58.829658 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:58.829793 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:58.829892 [ 31997 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:58.830320 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.830664 [ 31979 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.830974 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.831298 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.831973 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:58.832125 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:58.832600 [ 31982 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:58.833143 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.834915 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:58.835103 [ 31997 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:58.835190 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:58.835291 [ 31997 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:58.836450 [ 31987 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 2933ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:58.837495 [ 31997 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 13ms, ratio: 0.025641028, points per minute: 1183.432)
2026.05.06 08:49:58.838495 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:58.838642 [ 31982 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:58.839511 [ 31997 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:58.840834 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:58.840911 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:58.841030 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:58.841143 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:58.845831 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:58.846017 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:58.846437 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:58.846936 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:58.847804 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:58.848160 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:58.848182 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:58.849152 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:58.852746 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:58.852893 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:58.853057 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:58.853337 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.853565 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.853726 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:58.853821 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:58.854028 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:58.854958 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:58.855084 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:58.855226 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:58.855479 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.855722 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.855873 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:58.855966 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:58.856159 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:58.857234 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:58.857376 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:58.857514 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:58.857784 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.857978 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.858133 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:58.858226 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:58.858413 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:58.859375 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:58.859496 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:58.859685 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:58.859933 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.860123 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.860273 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:58.860365 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:58.860578 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:58.861639 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:58.861763 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:58.861903 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:58.862153 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.862353 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.862505 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:58.862613 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:58.862805 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:58.863808 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:58.863935 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:58.864076 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:58.864328 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.864541 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.864704 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:58.864793 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:58.864982 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:58.865720 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:58.865836 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:58.865967 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:58.866210 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.866404 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:49:58.866565 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:58.866655 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:58.866841 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:58.867345 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:58.868045 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:58.868441 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:58.868550 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:58.871860 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:58.873001 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:58.873232 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:58.873484 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:58.873858 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:58.874433 [ 31975 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:58.874752 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:58.874848 [ 31975 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:58.875081 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:58.876446 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:58.877856 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:58.878542 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:58.878627 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:58.878698 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:58.881332 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:58.883582 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:58.883747 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:58.887658 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:58.888061 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:58.888122 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:58.888218 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:58.888310 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:58.891926 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:58.892053 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:58.892446 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:58.892942 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:58.893827 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:58.894106 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:58.896849 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:58.896959 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:58.897067 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:58.897265 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.897412 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.897536 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:58.897604 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:58.897743 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:58.898314 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:58.898388 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:58.898473 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:58.898653 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.898789 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.898885 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:58.898939 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:58.899044 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:58.899802 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:58.899893 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:58.899990 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:58.900156 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.900285 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.900397 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:58.900460 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:58.900594 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:58.901241 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:58.901330 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:58.901426 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:58.901587 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.901709 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.901803 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:58.901855 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:58.901953 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:58.902627 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:58.902715 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:58.902811 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:58.902963 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.903102 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.903206 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:58.903271 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:58.903393 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:58.904050 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:58.904141 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:58.904237 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:58.904372 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.904480 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.904576 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:58.904631 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:58.904745 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:58.905193 [ 32282 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:58.905279 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:58.905374 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:58.905560 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.905690 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32282, kind: Regular, unbound: 0
2026.05.06 08:49:58.905786 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:58.905845 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:58.905963 [ 32282 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:58.906337 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:58.906798 [ 32282 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:58.907019 [ 32282 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:58.907068 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:58.910619 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:58.912002 [ 32282 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:58.912679 [ 32282 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:58.913090 [ 31977 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:58.913269 [ 31977 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:59.210885 [ 31971 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.211289 [ 31971 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.211335 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:59.211664 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.211832 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.211951 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.214577 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.214629 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.216536 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.216578 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.216866 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:59.216998 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:59.217109 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:59.217339 [ 31967 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.217850 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:59.217861 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:59.217922 [ 31967 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.218012 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:59.218123 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:59.218195 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.218249 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.218514 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.218731 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:59.219902 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.220236 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:59.220360 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.220993 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:59.221180 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:59.221547 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:59.221620 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.221999 [ 31966 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.221997 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:59.222407 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:59.222777 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.223010 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.223370 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:59.223502 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:59.223657 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:59.224785 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.226613 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.226965 [ 31961 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.227378 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.227451 [ 31961 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.227754 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:59.227771 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.228050 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.228761 [ 31986 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:59.228858 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:59.228996 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:59.229675 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.229761 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.231234 [ 31986 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 392ms, ratio: 0.025641028, points per minute: 39.24647)
2026.05.06 08:49:59.231285 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.231386 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:59.231403 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:59.231546 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:59.232182 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:59.232265 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:59.232287 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:59.232350 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:59.232403 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:59.232510 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:59.232807 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:59.234168 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:59.234852 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:59.235021 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:59.235309 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:59.235379 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:59.235742 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:59.236960 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:59.237074 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:59.237186 [ 31986 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:59.237597 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:59.237747 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:59.241793 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:59.243286 [ 31986 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:59.243375 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:59.243477 [ 31986 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:59.244451 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:59.244551 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:59.244678 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:59.244797 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:59.245863 [ 31986 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:49:59.247733 [ 31986 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 1ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:59.249720 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:59.249950 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:59.250362 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:59.250928 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:59.251894 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:59.252299 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:59.257092 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:59.257243 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:59.257395 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:59.257718 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.257937 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.258100 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:59.258198 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:59.258435 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:59.259393 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:59.259542 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:59.259728 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:59.259994 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.260198 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.260355 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:59.260448 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:59.260662 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:59.261751 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:59.261897 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:59.262039 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:59.262292 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.262490 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.262700 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:59.262800 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:59.262991 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:59.264079 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:59.264215 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:59.264357 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:59.264630 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.264825 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.264981 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:59.265074 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:59.265260 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:59.266355 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:59.266484 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:59.266647 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:59.266901 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.267106 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.267261 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:59.267356 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:59.267563 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:59.268572 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:59.268700 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:59.268842 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:59.269095 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.269290 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.269470 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:59.269602 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:59.269800 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:59.270565 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:59.270688 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:59.270822 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:59.271069 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.271264 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.271409 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:59.271499 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:59.271792 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:59.272327 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:59.273004 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:59.273408 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:59.273499 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:59.276481 [ 31949 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1750ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:59.277141 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:59.278626 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:59.279297 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:59.279775 [ 31952 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.280067 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.280117 [ 31952 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:59.280359 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.282058 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.283694 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:59.284391 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:59.284460 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:59.284534 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:59.287190 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:59.289500 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:59.289657 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:59.293606 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:59.293790 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:59.293841 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:59.293927 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:59.294003 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:59.297376 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:59.297489 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:59.297993 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:59.298545 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:59.299547 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:59.299810 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:59.302339 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:59.302435 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:59.302547 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:59.302729 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.302871 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.302969 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:59.303031 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:59.303157 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:59.303764 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:59.303848 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:59.303936 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:59.304089 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.304208 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.304302 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:59.304359 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:59.304469 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:59.305165 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:59.305251 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:59.305340 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:59.305486 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.305616 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.305714 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:59.305772 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:59.305879 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:59.306482 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:59.306576 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:59.306664 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:59.306804 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.306914 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.307003 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:59.307057 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:59.307165 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:59.307845 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:59.307924 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:59.308011 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:59.308158 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.308273 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.308362 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:59.308420 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:59.308542 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:59.309127 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:59.309206 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:59.309292 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:59.309438 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.309564 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.309658 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:59.309722 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:59.309831 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:59.310276 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:59.310349 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:59.310432 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:59.310590 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.310706 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:49:59.310793 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:59.310845 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:59.310951 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:59.311278 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:59.311687 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:59.311938 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:59.311992 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:49:59.315247 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:49:59.315911 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:59.316233 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:59.316666 [ 31954 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:59.316849 [ 31954 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:59.869201 [ 31947 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 4440ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:59.889417 [ 31965 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.889819 [ 31965 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.889858 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:59.890101 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.890278 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.890395 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.892841 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.892889 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.894456 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.894483 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.894726 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:49:59.894883 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:49:59.894985 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:59.895054 [ 31964 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.895478 [ 31964 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.895543 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:59.895749 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:59.895804 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.895840 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.895867 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:59.895975 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:49:59.896050 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.896614 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:59.897488 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.897571 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.897960 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:49:59.898642 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:59.898870 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:59.899163 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.899170 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:59.899218 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.899540 [ 31938 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.899716 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:59.899935 [ 31938 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.899955 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:59.900216 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.900304 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.900545 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.900986 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:49:59.901132 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:59.901264 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:49:59.902002 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.902056 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.903554 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.903644 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.904137 [ 31935 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.904466 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:49:59.904541 [ 31935 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.904788 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.904830 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.905083 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.906209 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.906398 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.907031 [ 31956 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:59.907134 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:49:59.907281 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:59.907563 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:59.907719 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.908076 [ 31934 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.908270 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:59.908379 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:59.908406 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.908479 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:59.908674 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.909469 [ 31956 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 662ms, ratio: 0.025641028, points per minute: 23.239601)
2026.05.06 08:49:59.909673 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:49:59.909765 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:49:59.910290 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.910331 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:49:59.910419 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:49:59.910540 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:49:59.911027 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:49:59.911140 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:59.911863 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.912188 [ 31924 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.912219 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:49:59.912514 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.912789 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.912793 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:49:59.913015 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:49:59.913326 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:49:59.913398 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:59.913547 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:49:59.913746 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:59.913938 [ 31932 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:49:59.914937 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:49:59.915036 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.915069 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:59.915175 [ 31956 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:49:59.916602 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.916964 [ 31926 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.917318 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.917445 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:59.917637 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.919415 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.919996 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:59.920585 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:59.920654 [ 31932 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:49:59.920703 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:59.920817 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:59.920971 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:49:59.921553 [ 31956 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:49:59.921648 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:49:59.921760 [ 31956 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:49:59.923717 [ 31956 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:49:59.925635 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:49:59.925822 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:49:59.926187 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:59.926231 [ 31956 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:49:59.926781 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:59.927806 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:49:59.928167 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:59.929826 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:49:59.930835 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:49:59.932641 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:49:59.932778 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:49:59.932919 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:59.933183 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.933388 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.933556 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:59.933650 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:59.933846 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:59.934731 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:49:59.934852 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:49:59.934988 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:59.935245 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.935436 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.935598 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:59.935720 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:59.935905 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:59.936934 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:49:59.937054 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:49:59.937187 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:59.937425 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.937629 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.937777 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:59.937869 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:59.938049 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:59.938975 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:49:59.939106 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:49:59.939238 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:59.939474 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.939706 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.939857 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:59.939946 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:59.940125 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:59.941149 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:49:59.941269 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:49:59.941405 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:59.941658 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.941852 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.942000 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:59.942089 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:59.942277 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:59.943210 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:49:59.943327 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:49:59.943461 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:59.943747 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.943937 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.944079 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:59.944165 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:59.944348 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:59.945061 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:49:59.945174 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:49:59.945301 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:59.945555 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.945742 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:49:59.945883 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:59.945982 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:59.946164 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:59.946662 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:49:59.947410 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:49:59.947861 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:49:59.947952 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:49:59.951819 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:49:59.953272 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:59.953970 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:59.954362 [ 32128 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:59.954660 [ 32128 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:49:59.954846 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:59.954964 [ 32128 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:49:59.955016 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:49:59.955334 [ 32275 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:49:59.955406 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:49:59.957354 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:49:59.959042 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:49:59.959889 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:49:59.960008 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:49:59.960113 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:49:59.962819 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:49:59.965083 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:49:59.965219 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:49:59.969083 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:49:59.971270 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:49:59.971348 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:49:59.971472 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:49:59.971636 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:49:59.976346 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:49:59.976550 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:49:59.976996 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:49:59.977631 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:49:59.978750 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:49:59.979214 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:49:59.983856 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:49:59.984005 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:49:59.984167 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:49:59.984481 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.984725 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.984890 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:49:59.984989 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:49:59.985196 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:49:59.986189 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:49:59.986326 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:49:59.986473 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:49:59.986756 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.986964 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.987121 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:49:59.987230 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:49:59.987428 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:49:59.988608 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:49:59.988743 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:49:59.988886 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:59.989138 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.989341 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.989499 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:59.989608 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:59.989801 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:59.990791 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:49:59.990919 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:49:59.991060 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:49:59.991308 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.991505 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.991731 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:49:59.991828 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:49:59.992018 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:49:59.993109 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:49:59.993237 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:49:59.993381 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:49:59.993656 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.993865 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.994020 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:49:59.994120 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:49:59.994336 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:49:59.995325 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:49:59.995449 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:49:59.995647 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:49:59.995904 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.996118 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.996272 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:49:59.996362 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:49:59.996568 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:49:59.997366 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:49:59.997494 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:49:59.997656 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:49:59.997913 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.998114 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:49:59.998263 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:49:59.998355 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:49:59.998564 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:49:59.999102 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:49:59.999907 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:00.000339 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:00.000432 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:00.004335 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:00.005853 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:00.006582 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:00.006994 [ 31912 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:00.007209 [ 31912 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:00.161947 [ 31930 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:00.166131 [ 31930 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:00.175662 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:00.176769 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:00.203712 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:00.204251 [ 32284 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0917
8. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003afe2f18
9. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
10. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
11. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
12. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
13. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
14. ? @ 0x00007f6e2fbe4ac3
15. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:00.397884 [ 31906 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:00.398335 [ 31906 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:00.398407 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:00.398695 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:00.398884 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:00.398980 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:00.401072 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:00.401203 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:00.402560 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:00.402674 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:00.402822 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:00.402977 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:00.403062 [ 31923 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:00.403093 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:00.403253 [ 31923 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:00.403328 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:00.403485 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:00.403646 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:00.403692 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:00.403802 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:00.403914 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:00.403933 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:00.404499 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:00.405139 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:00.405554 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:00.405954 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:00.406547 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:00.406726 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:00.406866 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:00.407062 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:00.407230 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:00.407283 [ 32116 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:00.407538 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:00.407685 [ 32116 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:00.407707 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:00.407845 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:00.408014 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:00.408353 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:00.408929 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:00.409057 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:00.409165 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:00.409732 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:00.409997 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:00.411339 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:00.411657 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:00.412052 [ 31901 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:00.412383 [ 31901 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:00.412404 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:00.412572 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:00.412682 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:00.412752 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:00.414129 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:00.414330 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:00.414706 [ 31909 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:00.414812 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:00.414988 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:00.415753 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:00.415967 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:00.416728 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:00.416857 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:00.416957 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:00.417347 [ 31909 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 491ms, ratio: 0.025641028, points per minute: 31.333231)
2026.05.06 08:50:00.417547 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:00.417638 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:00.418209 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:00.418294 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:00.418385 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:00.418861 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:00.419952 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:00.420122 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:00.420773 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:00.420943 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:00.421239 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:00.421818 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:00.422319 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:00.422489 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:00.422980 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:00.423094 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:00.423194 [ 31909 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:00.426814 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:00.429192 [ 31909 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:00.429307 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:00.429441 [ 31909 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:00.429893 [ 32113 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:00.429951 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:00.430054 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:00.430201 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:00.430326 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:00.431772 [ 31909 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:50:00.435289 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:00.435466 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:00.435877 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:00.436481 [ 32113 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:00.436504 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:00.437536 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:00.437890 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:00.441068 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:00.441620 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:00.442384 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:00.442539 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:00.442687 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:00.442957 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.443166 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.443321 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:00.443415 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:00.443686 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:00.444647 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:00.444770 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:00.444907 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:00.445155 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.445353 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.445502 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:00.445612 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:00.445800 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:00.446829 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:00.446950 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:00.447083 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:00.447321 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.447508 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.447720 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:00.447814 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:00.447999 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:00.448936 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:00.449053 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:00.449184 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:00.449423 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.449622 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.449769 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:00.449860 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:00.450042 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:00.451068 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:00.451188 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:00.451324 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:00.451580 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.451859 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.452011 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:00.452104 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:00.452293 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:00.453326 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:00.453451 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:00.453606 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:00.453862 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.453874 [ 32331 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:00.454001 [ 32331 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:00.454052 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.454197 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:00.454285 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:00.454289 [ 32331 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.004957857 sec. (201.700 rows/sec., 4.73 KiB/sec.)
2026.05.06 08:50:00.454401 [ 32331 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:00.454471 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:00.455223 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:00.455352 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:00.455515 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:00.455837 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.456062 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:00.456209 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:00.456298 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:00.456486 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:00.457014 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:00.457118 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:00.457427 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:00.457697 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:00.458092 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:00.458181 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:00.460452 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:00.460910 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:00.461190 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:00.462483 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:00.463125 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:00.463560 [ 32112 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:00.463897 [ 32112 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:00.464218 [ 32112 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:00.464276 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:00.464687 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:00.466779 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:00.466815 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:00.468832 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:00.469006 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:00.469162 [ 32280 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:00.469716 [ 32110 ] {} <Debug> test_1.t1 (bed21b2e-93a1-4da1-ba17-fc452427a4b6): According to exponential backoff policy, do not perform mutations for the part 1_1_1_0 yet. Put it aside.
2026.05.06 08:50:00.470007 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:00.470125 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:00.470223 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:00.473071 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:00.475268 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:00.475409 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:00.479708 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:00.482080 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:00.482153 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:00.482270 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:00.482381 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:00.487038 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:00.487222 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:00.487690 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:00.488212 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:00.489234 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:00.489624 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:00.494109 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:00.494254 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:00.494404 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:00.494699 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.494914 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.495074 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:00.495183 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:00.495388 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:00.496373 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:00.496507 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:00.496675 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:00.496937 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.497145 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.497304 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:00.497398 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:00.497636 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:00.498691 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:00.498818 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:00.498959 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:00.499213 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.499413 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.500022 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:00.500124 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:00.500321 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:00.501293 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:00.501420 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:00.501577 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:00.501835 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.502032 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.502189 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:00.502282 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:00.502473 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:00.503552 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:00.503736 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:00.503886 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:00.504145 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.504366 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.504542 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:00.504638 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:00.504839 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:00.505788 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:00.505914 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:00.506056 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:00.506312 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.506513 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.506685 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:00.506775 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:00.506969 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:00.507769 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:00.507897 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:00.508051 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:00.508313 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.508535 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:00.508685 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:00.508776 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:00.508972 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:00.509469 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:00.510105 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:00.510495 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:00.510602 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:00.515173 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:00.516544 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:00.517183 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:00.517613 [ 32111 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:00.745456 [ 31889 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 2474ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:01.165735 [ 31878 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1889ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:01.367968 [ 31876 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 8156ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:01.392260 [ 32115 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:01.392722 [ 32115 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:01.392742 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:01.393054 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:01.393073 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:01.393391 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:01.395183 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:01.395283 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:01.396952 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:01.397043 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:01.397183 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:01.397314 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:01.397417 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:01.397472 [ 31875 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:01.397904 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:01.397983 [ 31875 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:01.398207 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:01.398228 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:01.398280 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:01.398325 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:01.398430 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:01.398595 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:01.399067 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:01.400305 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:01.400365 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:01.400636 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:01.401367 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:01.401630 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:01.401907 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:01.402078 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:01.402183 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:01.402466 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:01.402658 [ 32103 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:01.403005 [ 32103 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:01.403006 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:01.403312 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:01.403344 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:01.403808 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:01.404114 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:01.404268 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:01.404408 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:01.405185 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:01.405745 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:01.406799 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:01.407121 [ 32093 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:01.407427 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:01.407775 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:01.407787 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:01.409635 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:01.410009 [ 31869 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:01.410110 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:01.410231 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:01.411229 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:01.412031 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:01.412140 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:01.412239 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:01.412890 [ 31869 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 982ms, ratio: 0.025641028, points per minute: 15.6666155)
2026.05.06 08:50:01.413105 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:01.413221 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:01.413863 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:01.413978 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:01.414096 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:01.414710 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:01.415390 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:01.416356 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:01.417165 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:01.417391 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:01.417810 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:01.417957 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:01.418098 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:01.418299 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:01.419780 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:01.419963 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:01.420099 [ 31869 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:01.422950 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:01.425195 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:01.425272 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:01.425391 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:01.425506 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:01.426414 [ 31869 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:01.426516 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:01.426729 [ 31869 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:01.430160 [ 31869 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:50:01.430274 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:01.430452 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:01.431017 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:01.431502 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:01.432652 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:01.433080 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:01.438202 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:01.438364 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:01.438515 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:01.438815 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.439057 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.439234 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:01.439334 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:01.439563 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:01.440554 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:01.440686 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:01.440829 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:01.441090 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.441292 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.441447 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:01.441558 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:01.441753 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:01.442755 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:01.442863 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:01.442982 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:01.443191 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.443357 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.443489 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:01.443586 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:01.443773 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:01.444612 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:01.444718 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:01.444837 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:01.445046 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.445209 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.445339 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:01.445428 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:01.445600 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:01.446495 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:01.446617 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:01.446737 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:01.446951 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.447122 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.447251 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:01.447332 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:01.447495 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:01.448334 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:01.448441 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:01.448578 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:01.448790 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.448954 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.449089 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:01.449164 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:01.449323 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:01.449957 [ 32272 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:01.450057 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:01.450172 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:01.450378 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.450558 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32272, kind: Regular, unbound: 0
2026.05.06 08:50:01.450680 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:01.450756 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:01.450799 [ 32081 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:01.450914 [ 32272 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:01.451363 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:01.452021 [ 32272 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:01.452362 [ 32272 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:01.452439 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:01.454862 [ 32081 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:01.455430 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:01.456971 [ 32272 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:01.457711 [ 32272 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:01.458253 [ 32087 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:01.458686 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:01.458717 [ 32087 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:01.459039 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:01.461542 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:01.463861 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:01.464318 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:01.464853 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:01.464972 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:01.465075 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:01.465354 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:01.467767 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:01.469806 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:01.469960 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:01.474291 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:01.476567 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:01.476644 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:01.476763 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:01.476874 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:01.481694 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:01.481878 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:01.482403 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:01.483061 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:01.484155 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:01.484573 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:01.489466 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:01.489663 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:01.489839 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:01.490127 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.490347 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.490507 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:01.490628 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:01.490835 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:01.491821 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:01.491957 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:01.492101 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:01.492253 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:01.492360 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.492577 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.492732 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:01.492839 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:01.492812 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:01.493033 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:01.494183 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:01.494319 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:01.494463 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:01.494804 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.495032 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.495192 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:01.495290 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:01.495492 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:01.496673 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:01.496814 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:01.496958 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:01.497217 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.497458 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.497706 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:01.497855 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:01.498125 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:01.499744 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:01.499956 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:01.500169 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:01.500553 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.500843 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.501061 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:01.501189 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:01.501448 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:01.502875 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:01.503063 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:01.503268 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:01.503710 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.504004 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.504237 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:01.504373 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:01.504673 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:01.505842 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:01.506023 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:01.506223 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:01.506609 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.506904 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:01.507131 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:01.507270 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:01.507567 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:01.508427 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:01.509412 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:01.509925 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:01.510036 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:01.513848 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:01.515125 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:01.515798 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:01.516104 [ 31860 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:01.590901 [ 31861 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 8374ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:01.769285 [ 31854 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:01.773655 [ 31854 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:01.783418 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:01.784648 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:01.807420 [ 32398 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:01.807598 [ 32398 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:01.807945 [ 32398 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007006159 sec. (142.732 rows/sec., 3.35 KiB/sec.)
2026.05.06 08:50:01.808038 [ 32398 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:01.812353 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:01.813034 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:01.820031 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:01.820793 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:01.832488 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:01.835550 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:01.836066 [ 32273 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:01.962072 [ 31850 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:01.968142 [ 31850 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:01.977105 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:01.978160 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:02.002984 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=4, jobs=4). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:02.003572 [ 32285 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=4, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:02.331845 [ 31839 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.332318 [ 31839 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.332402 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:02.332589 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:02.332793 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.332798 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.335348 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.335444 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.337184 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.337274 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.337433 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:02.337630 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:02.337739 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:02.337756 [ 31840 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.338082 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:02.338162 [ 31840 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.338387 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.338501 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:02.338737 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:02.338856 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:02.338947 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.338967 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:02.339595 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:02.340891 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.341584 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.341674 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:02.342972 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:02.343294 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:02.343723 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.343885 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:02.344322 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:02.344448 [ 31836 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.344929 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:02.345340 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.346012 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.346223 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:02.346431 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:02.346615 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:02.347857 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.350270 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.350773 [ 31830 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.351201 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:02.351282 [ 31830 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.351654 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:02.351672 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.351938 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.353378 [ 31851 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:02.353483 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:02.353632 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:02.353968 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.354071 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.356515 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.356535 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:02.356594 [ 31851 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 926ms, ratio: 0.025641028, points per minute: 16.614058)
2026.05.06 08:50:02.356829 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:02.356933 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:02.357570 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:02.357664 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:02.357670 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:02.357823 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:02.357835 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:02.357958 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:02.358400 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:02.360507 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:02.360765 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:02.361580 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:02.361810 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:02.362256 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:02.362803 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:02.363054 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:02.363207 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:02.364543 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:02.364720 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:02.364840 [ 31851 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:02.367251 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:02.369652 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:02.369753 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:02.369888 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:02.369996 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:02.370302 [ 31851 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:02.370419 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:02.370567 [ 31851 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:02.373824 [ 31851 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 17ms, ratio: 0.025641028, points per minute: 904.9774)
2026.05.06 08:50:02.374907 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:02.375074 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:02.375483 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:02.376011 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:02.377014 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:02.377365 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:02.381769 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:02.381923 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:02.382077 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:02.382365 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.382600 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.382764 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:02.382882 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:02.383110 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:02.384180 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:02.384322 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:02.384473 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:02.384771 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.384992 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.385154 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:02.385251 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:02.385474 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:02.386735 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:02.386913 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:02.387081 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:02.387358 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.387585 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.387789 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:02.387933 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:02.388144 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:02.389249 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:02.389386 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:02.389552 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:02.389841 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.390087 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.390270 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:02.390368 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:02.390585 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:02.391804 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:02.391991 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:02.392150 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:02.392427 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.392652 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.392831 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:02.392971 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:02.393189 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:02.394350 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:02.394539 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:02.394718 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:02.395071 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.395343 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.395591 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:02.395838 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:02.396100 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:02.397133 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:02.397274 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:02.397422 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:02.397711 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.397960 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:02.398128 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:02.398225 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:02.398426 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:02.399045 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:02.399806 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:02.400275 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:02.400391 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:02.404813 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:02.406154 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:02.406834 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:02.407288 [ 31841 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:02.407660 [ 31841 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.408059 [ 31841 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:02.408107 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:02.408480 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.410920 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.413339 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:02.414405 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:02.414550 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:02.414658 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:02.417695 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:02.420270 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:02.420455 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:02.425063 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:02.427301 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:02.427393 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:02.427547 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:02.427702 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:02.433159 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:02.433379 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:02.434020 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:02.434627 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:02.435850 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:02.436384 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:02.443698 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:02.443953 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:02.444185 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:02.444636 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.444963 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.445189 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:02.445309 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:02.445556 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:02.446876 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:02.447078 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:02.447302 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:02.447777 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.448098 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.448315 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:02.448454 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:02.448775 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:02.450492 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:02.450725 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:02.450953 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:02.451338 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.451712 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.451960 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:02.452083 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:02.452326 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:02.453764 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:02.453974 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:02.454187 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:02.454599 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.454920 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.455163 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:02.455279 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:02.455514 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:02.457182 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:02.457391 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:02.457589 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:02.457948 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.458196 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.458410 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:02.458562 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:02.458810 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:02.460257 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:02.460447 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:02.460678 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:02.461031 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.461308 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.461539 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:02.461685 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:02.461976 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:02.463081 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:02.463264 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:02.463437 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:02.463920 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.464224 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:02.464420 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:02.464536 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:02.464774 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:02.465480 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:02.466245 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:02.466740 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:02.466890 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:02.472161 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:02.473780 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:02.474642 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:02.475210 [ 31820 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:02.475545 [ 31820 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:02.667183 [ 31815 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.667643 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:02.667661 [ 31815 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.667902 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.668013 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:02.668418 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.670340 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.670379 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.672563 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.672627 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.672819 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:02.672953 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:02.673068 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:02.673138 [ 31829 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.673579 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:02.673641 [ 31829 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.673869 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:02.673903 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.673963 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:02.673985 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:02.674150 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:02.674314 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.674827 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:02.676078 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.676153 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.676332 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:02.677114 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:02.677394 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:02.677886 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:02.678244 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.678238 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.678296 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:02.678645 [ 31826 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.678952 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:02.679022 [ 31826 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.679246 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:02.679360 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.679397 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.679587 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:02.679681 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:02.679741 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:02.681209 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.681278 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.683011 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.683103 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.683258 [ 31818 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:02.683362 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:02.683489 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:02.683589 [ 31813 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.684015 [ 31813 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.684029 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:02.684374 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.684463 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:02.684840 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.686331 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.686361 [ 31818 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 313ms, ratio: 0.025641028, points per minute: 49.15213)
2026.05.06 08:50:02.686619 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:02.686752 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:02.686925 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.687387 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:02.687501 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:02.687651 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:02.688190 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.688212 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:02.688675 [ 31807 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.688838 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:02.689000 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:02.689280 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.689779 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:02.689783 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:02.689992 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:02.690103 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:02.690483 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:02.690692 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:02.690974 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.690996 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:02.691397 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:02.692696 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:02.692835 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:02.692852 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:02.692948 [ 31818 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:02.693272 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:02.695119 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:02.695291 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:02.698401 [ 31818 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:02.698506 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:02.698654 [ 31818 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:02.700955 [ 31818 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:50:02.701695 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:02.703001 [ 31818 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:02.703996 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:02.704074 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:02.704194 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:02.704309 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:02.708750 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:02.708935 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:02.709387 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:02.709867 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:02.710922 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:02.711346 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:02.716329 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:02.716482 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:02.716655 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:02.716982 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.717222 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.717386 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:02.717486 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:02.717720 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:02.718736 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:02.718887 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:02.719047 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:02.719318 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.719543 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.719746 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:02.719858 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:02.720073 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:02.721373 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:02.721552 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:02.721747 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:02.722040 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.722247 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.722413 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:02.722507 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:02.722722 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:02.723798 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:02.723931 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:02.724074 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:02.724334 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.724558 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.724719 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:02.724817 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:02.725012 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:02.726244 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:02.726390 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:02.726561 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:02.726830 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.727040 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.727200 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:02.727299 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:02.727531 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:02.728641 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:02.728777 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:02.728925 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:02.729240 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.729497 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.729670 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:02.729762 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:02.729959 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:02.730848 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:02.730980 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:02.731121 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:02.731376 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.731724 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:02.731879 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:02.731973 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:02.732169 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:02.732727 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:02.733397 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:02.733825 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:02.733918 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:02.738169 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:02.739503 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:02.740209 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:02.740681 [ 31804 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:02.741069 [ 31804 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:02.741119 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:02.741448 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:02.743260 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:02.744979 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:02.745767 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:02.745892 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:02.745995 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:02.748903 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:02.751365 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:02.751545 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:02.755994 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:02.758687 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:02.758772 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:02.758899 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:02.759038 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:02.764034 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:02.764222 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:02.764669 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:02.765117 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:02.766085 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:02.766472 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:02.771418 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:02.771640 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:02.771809 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:02.772097 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.772335 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.772510 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:02.772633 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:02.772841 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:02.773795 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:02.773925 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:02.774069 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:02.774326 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.774549 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.774706 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:02.774801 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:02.774996 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:02.776169 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:02.776304 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:02.776446 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:02.776724 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.776937 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.777095 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:02.777192 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:02.777382 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:02.778382 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:02.778507 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:02.778666 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:02.778919 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.779114 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.779268 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:02.779361 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:02.779565 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:02.780840 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:02.780983 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:02.781152 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:02.781428 [ 31795 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1616ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:02.781458 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.781716 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.781880 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:02.781979 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:02.782180 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:02.783175 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:02.783304 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:02.783448 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:02.783749 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.783950 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.784134 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:02.784245 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:02.784447 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:02.785251 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:02.785372 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:02.785511 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:02.785783 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.785981 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:02.786131 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:02.786221 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:02.786413 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:02.786960 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:02.787670 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:02.788079 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:02.788170 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:02.792080 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:02.793604 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:02.794313 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:02.794714 [ 31792 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:02.794883 [ 31792 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:02.985849 [ 31788 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:02.993767 [ 31788 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:03.002855 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:03.003928 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:03.026664 [ 32403 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:03.026807 [ 32403 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:03.027076 [ 32403 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006981188 sec. (143.242 rows/sec., 3.36 KiB/sec.)
2026.05.06 08:50:03.027152 [ 32403 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:03.031994 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:03.032793 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:03.037715 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:03.038756 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:03.051694 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:03.055781 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:03.056506 [ 32276 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:03.679084 [ 31761 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:03.687270 [ 31761 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:03.696800 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:03.697888 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:03.723373 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=7, jobs=7). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:03.723962 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=7, jobs=7). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:03.756149 [ 31771 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 3011ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:03.810599 [ 31753 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:03.811008 [ 31753 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:03.811135 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:03.811417 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:03.811592 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:03.811896 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:03.814675 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:03.814747 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:03.817076 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:03.817148 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:03.817362 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:03.817504 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:03.817632 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:03.817841 [ 31749 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:03.818226 [ 31749 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:03.818243 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:03.818297 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:03.818399 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:03.818513 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:03.818562 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:03.818602 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:03.818906 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:03.819138 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:03.820554 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:03.820825 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:03.820898 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:03.821550 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:03.821741 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:03.822136 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:03.822264 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:03.822627 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:03.822666 [ 31754 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:03.823031 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:03.823365 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:03.823655 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:03.823992 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:03.824124 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:03.824117 [ 31746 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:03.824241 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:03.824481 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:03.824755 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:03.825191 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:03.826400 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:03.826857 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:03.828141 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:03.828481 [ 31744 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:03.828834 [ 31744 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:03.828885 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:03.829138 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:03.829235 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:03.829402 [ 31752 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:03.829501 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:03.829501 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:03.829673 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:03.831130 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:03.831252 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:03.832088 [ 31752 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1129ms, ratio: 0.025641028, points per minute: 13.626764)
2026.05.06 08:50:03.832277 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:03.832374 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:03.832804 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:03.832823 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:03.832990 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:03.833106 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:03.833210 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:03.833635 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:03.833741 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:03.833746 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:03.833909 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:03.835077 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:03.835834 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:03.836016 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:03.836334 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:03.836780 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:03.836973 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:03.838016 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:03.838152 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:03.838261 [ 31752 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:03.839471 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:03.839733 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:03.844984 [ 31752 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:03.845086 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:03.845208 [ 31752 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:03.846569 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:03.847989 [ 31752 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:50:03.848919 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:03.848993 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:03.849128 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:03.849242 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:03.850219 [ 31752 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:03.853988 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:03.854175 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:03.854666 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:03.855202 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:03.856142 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:03.856511 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:03.861050 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:03.861196 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:03.861347 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:03.861645 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.861862 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.862025 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:03.862124 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:03.862332 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:03.863276 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:03.863422 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:03.863581 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:03.863903 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.864112 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.864270 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:03.864365 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:03.864586 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:03.865679 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:03.865810 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:03.865951 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:03.866206 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.866405 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.866577 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:03.866673 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:03.866867 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:03.867902 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:03.868032 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:03.868175 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:03.868431 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.868644 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.868800 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:03.868895 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:03.869107 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:03.870280 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:03.870417 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:03.870585 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:03.870850 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.871062 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.871221 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:03.871335 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:03.871556 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:03.872630 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:03.872763 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:03.872909 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:03.873171 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.873384 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.873555 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:03.873649 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:03.873847 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:03.874612 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:03.874735 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:03.874873 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:03.875122 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.875319 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:03.875484 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:03.875592 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:03.875816 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:03.876349 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:03.877041 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:03.877448 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:03.877556 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:03.880992 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:03.882319 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:03.882969 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:03.883436 [ 31735 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:03.883890 [ 31735 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:03.884269 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:03.884303 [ 31735 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:03.884591 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:03.886274 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:03.888007 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:03.888780 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:03.888887 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:03.888986 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:03.891791 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:03.894066 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:03.894213 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:03.898223 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:03.900348 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:03.900445 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:03.900582 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:03.900697 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:03.905302 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:03.905487 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:03.905989 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:03.906619 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:03.907662 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:03.908041 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:03.912602 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:03.912749 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:03.912894 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:03.913169 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.913391 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.913599 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:03.913698 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:03.913914 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:03.914874 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:03.915018 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:03.915161 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:03.915426 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.915686 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.915847 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:03.915941 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:03.916141 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:03.917219 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:03.917347 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:03.917487 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:03.917756 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.917954 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.918109 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:03.918202 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:03.918393 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:03.919397 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:03.919536 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:03.919716 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:03.920019 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.920251 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.920416 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:03.920512 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:03.920725 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:03.921847 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:03.921975 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:03.922120 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:03.922355 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.922539 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.922669 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:03.922749 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:03.922924 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:03.923779 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:03.923889 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:03.924010 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:03.924222 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.924387 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.924513 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:03.924600 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:03.924760 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:03.925378 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:03.925478 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:03.925606 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:03.925812 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.925974 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:03.926130 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:03.926205 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:03.926362 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:03.926815 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:03.927429 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:03.927837 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:03.927917 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:03.932429 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:03.933528 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:03.934054 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:03.934489 [ 31731 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:03.934706 [ 31731 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:03.954004 [ 31732 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1173ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:04.168385 [ 31727 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:04.168836 [ 31727 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:04.168892 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:04.169207 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:04.169236 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:04.169638 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:04.171324 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:04.171435 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:04.172859 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:04.173078 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:04.173120 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:04.173304 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:04.173425 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:04.173428 [ 31725 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:04.173958 [ 31725 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:04.173961 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:04.174081 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:04.174187 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:04.174267 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:04.174303 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:04.174272 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:04.174627 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:04.174963 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:04.175880 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:04.176232 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:04.176284 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:04.176854 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:04.177069 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:04.177382 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:04.177423 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:04.177866 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:04.178742 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:04.178993 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:04.179074 [ 31718 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:04.179149 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:04.179262 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:04.179430 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:04.179497 [ 31718 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:04.179736 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:04.179816 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:04.180084 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:04.181071 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:04.181487 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:04.182269 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:04.182880 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:04.183219 [ 31711 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:04.183592 [ 31711 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:04.183642 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:04.183924 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:04.183918 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:04.184292 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:04.184507 [ 31722 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:04.184640 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:04.184796 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:04.185776 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:04.186072 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:04.186915 [ 31722 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 337ms, ratio: 0.025641028, points per minute: 45.65168)
2026.05.06 08:50:04.187092 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:04.187127 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:04.187183 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:04.187487 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:04.187764 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:04.187859 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:04.187859 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:04.187939 [ 31750 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:04.187994 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:04.187957 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:04.188104 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:04.188296 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:04.188497 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:04.188601 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:04.189726 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:04.190101 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:04.190299 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:04.190495 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:04.190794 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:04.190841 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:04.191214 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:04.192387 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:04.192516 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:04.192541 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:04.192637 [ 31722 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:04.193397 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:04.193565 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:04.197543 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:04.199137 [ 31722 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:04.199237 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:04.199358 [ 31722 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:04.199841 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:04.199917 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:04.200039 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:04.200193 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:04.202719 [ 31722 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:50:04.205394 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:04.205598 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:04.205873 [ 31722 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 4ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:04.206201 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:04.206898 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:04.208013 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:04.208390 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:04.213021 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:04.213167 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:04.213366 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:04.213688 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.213905 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.214065 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:04.214162 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:04.214379 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:04.215334 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:04.215465 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:04.215655 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:04.215912 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.216126 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.216279 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:04.216372 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:04.216638 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:04.217824 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:04.217954 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:04.218092 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:04.218369 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.218581 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.218771 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:04.218867 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:04.219057 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:04.220265 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:04.220432 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:04.220643 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:04.220923 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.221145 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.221300 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:04.221394 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:04.221598 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:04.222722 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:04.222852 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:04.222995 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:04.223248 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.223455 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.223653 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:04.223753 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:04.223948 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:04.224915 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:04.225097 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:04.225242 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:04.225495 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.225706 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.225856 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:04.225958 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:04.226152 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:04.226926 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:04.227048 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:04.227182 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:04.227441 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.227691 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:04.227843 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:04.227932 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:04.228123 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:04.228671 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:04.229323 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:04.229737 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:04.229828 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:04.233617 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:04.234926 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:04.235574 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:04.236082 [ 31710 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:04.236515 [ 31710 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:04.236573 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:04.236906 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:04.238747 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:04.240449 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:04.241248 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:04.241360 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:04.241465 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:04.244319 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:04.246620 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:04.246767 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:04.250791 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:04.253150 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:04.253235 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:04.253360 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:04.253475 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:04.258411 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:04.258619 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:04.259124 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:04.259790 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:04.260846 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:04.261254 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:04.266226 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:04.266390 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:04.266569 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:04.266869 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.267096 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.267262 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:04.267363 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:04.267598 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:04.268644 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:04.268778 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:04.268927 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:04.269266 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.269560 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.269778 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:04.269897 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:04.270134 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:04.271653 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:04.271830 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:04.272011 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:04.272329 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.272599 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.272799 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:04.272920 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:04.273157 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:04.274468 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:04.274650 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:04.274821 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:04.275118 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.275374 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.275591 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:04.275760 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:04.275994 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:04.277378 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:04.277563 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:04.277744 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:04.278066 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.278323 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.278537 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:04.278659 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:04.278904 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:04.280280 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:04.280454 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:04.280740 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:04.281065 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.281317 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.281511 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:04.281654 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:04.281894 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:04.282877 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:04.283043 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:04.283214 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:04.283550 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.283832 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:04.284022 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:04.284141 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:04.284378 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:04.285088 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:04.285991 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:04.286547 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:04.286668 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:04.291155 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:04.292754 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:04.293584 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:04.294116 [ 31702 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:04.294325 [ 31702 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:04.641073 [ 31715 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 3050ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:04.672820 [ 31701 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:04.679282 [ 31701 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:04.688563 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:04.689606 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:04.711514 [ 32442 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:04.711706 [ 32442 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:04.712000 [ 32442 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006671868 sec. (149.883 rows/sec., 3.51 KiB/sec.)
2026.05.06 08:50:04.712085 [ 32442 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:04.715994 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:04.716639 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:04.723440 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:04.724189 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:04.736316 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:04.739204 [ 32279 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:04.739735 [ 32279 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:05.050780 [ 31693 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1096ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:05.062141 [ 31690 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:05.068278 [ 31690 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:05.078937 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:05.080555 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:05.096594 [ 31640 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #5
2026.05.06 08:50:05.096682 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:50:05.097079 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:05.110566 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:05.111137 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: failed to start the thread (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./build_docker/./src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e36eb
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:405: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000003afde6d5
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003afde228
9. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003b01e6bd
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:05.212015 [ 31688 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:05.212423 [ 31688 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:05.212459 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:05.212818 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:05.212854 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:05.213187 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:05.215656 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:05.215747 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:05.217581 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:05.217730 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:05.217828 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:05.217968 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:05.218071 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:05.218136 [ 31687 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:05.218619 [ 31687 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:05.218697 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:05.218922 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:05.218951 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:05.219078 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:05.219127 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:05.219217 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:05.219220 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:05.219930 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:05.221176 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:05.221251 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:05.221502 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:05.222366 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:05.222640 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:05.222995 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:05.223087 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:05.223229 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:05.223471 [ 31685 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:05.223494 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:05.223886 [ 31685 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:05.223908 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:05.224199 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:05.224212 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:05.224511 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:05.225272 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:05.225447 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:05.225633 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:05.227186 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:05.227245 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:05.229335 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:05.229389 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:05.229952 [ 31900 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:05.230370 [ 31900 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:05.230375 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:05.230690 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:05.230696 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:05.231012 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:05.231932 [ 31913 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:05.232053 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:05.232251 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:05.232503 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:05.232740 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:05.234475 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:05.234677 [ 31913 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1028ms, ratio: 0.025641028, points per minute: 14.965581)
2026.05.06 08:50:05.234704 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:05.234835 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:05.234917 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:05.235358 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:05.235447 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:05.235547 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:05.235552 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:05.235726 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:05.235841 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:05.236031 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:05.237444 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:05.237816 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:05.238463 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:05.238561 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:05.238584 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:05.238756 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:05.239121 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:05.239650 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:05.240777 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:05.240793 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:05.240897 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:05.240952 [ 31913 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:05.243233 [ 31913 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:05.243285 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:05.243352 [ 31913 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:05.243391 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:05.243462 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:05.243636 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:05.243761 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:05.245241 [ 31913 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 11ms, ratio: 0.025641028, points per minute: 1398.6014)
2026.05.06 08:50:05.248894 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:05.249080 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:05.249504 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:05.250069 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:05.250994 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:05.251369 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:05.255901 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:05.256047 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:05.256198 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:05.256488 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.256717 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.256876 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:05.256973 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:05.257181 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:05.258112 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:05.258238 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:05.258381 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:05.258651 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.258855 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.259028 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:05.259126 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:05.259321 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:05.260460 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:05.260622 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:05.260763 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:05.261014 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.261209 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.261365 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:05.261460 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:05.261665 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:05.262643 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:05.262770 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:05.262910 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:05.263163 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.263360 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.263514 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:05.263657 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:05.263850 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:05.264947 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:05.265075 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:05.265219 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:05.265472 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.265696 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.265852 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:05.265948 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:05.266145 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:05.267117 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:05.267241 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:05.267362 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:05.267580 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.267768 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.267896 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:05.267981 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:05.268140 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:05.268780 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:05.268880 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:05.268993 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:05.269198 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.269358 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:05.269480 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:05.269569 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:05.269727 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:05.270167 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:05.270753 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:05.271093 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:05.271169 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:05.274769 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:05.276266 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:05.276961 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:05.277384 [ 32078 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:05.277675 [ 32078 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:05.278002 [ 32078 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:05.278057 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:05.278376 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:05.280207 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:05.281920 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:05.282720 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:05.282829 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:05.282929 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:05.286074 [ 32542 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:47336, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:05.288624 [ 32542 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:05.288796 [ 32542 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:05.293306 [ 32542 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:05.295717 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:05.295798 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:05.295920 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:05.296034 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:05.300781 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:05.300962 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:05.301408 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:05.302022 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:05.302941 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:05.303340 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:05.308110 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:05.308255 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:05.308403 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:05.308696 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.308922 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.309080 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:05.309177 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:05.309379 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:05.310372 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:05.310503 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:05.310663 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:05.310916 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.311118 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.311273 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:05.311366 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:05.311642 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:05.312727 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:05.312895 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:05.313038 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:05.313300 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.313515 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.313692 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:05.313787 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:05.314004 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:05.315016 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:05.315183 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:05.315339 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:05.315663 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.315871 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.316026 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:05.316118 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:05.316306 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:05.317393 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:05.317539 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:05.317689 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:05.317970 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.318177 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.318333 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:05.318427 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:05.318639 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:05.319638 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:05.319776 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:05.319917 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:05.320169 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.320363 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.320511 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:05.320627 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:05.320815 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:05.321609 [ 32285 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:05.321733 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:05.321868 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:05.322114 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.322307 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32285, kind: Regular, unbound: 0
2026.05.06 08:50:05.322453 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:05.322556 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:05.322744 [ 32285 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:05.323249 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:05.323978 [ 32285 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:05.324422 [ 32285 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:05.324538 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:05.327858 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:05.329146 [ 32285 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:05.329816 [ 32285 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:05.330206 [ 31865 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:05.330382 [ 31865 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:05.995827 [ 32074 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:06.002070 [ 32074 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:06.011041 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:06.012193 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:06.035565 [ 32450 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:06.035737 [ 32450 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:06.036050 [ 32450 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006707488 sec. (149.087 rows/sec., 3.49 KiB/sec.)
2026.05.06 08:50:06.036138 [ 32450 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:06.040229 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:06.040916 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:06.049075 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:06.049716 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:06.061858 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:06.065452 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:06.065950 [ 32280 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:06.349408 [ 32065 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 10610ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:06.359089 [ 31899 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1309ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:06.409055 [ 32069 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:06.409474 [ 32069 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:06.409653 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:06.409859 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:06.410108 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:06.410266 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:06.412539 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:06.412617 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:06.414339 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:06.414416 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:06.414623 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:06.414764 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:06.414868 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:06.414979 [ 31873 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:06.415307 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:06.415358 [ 31873 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:06.415704 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:06.415713 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:06.415780 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:06.415915 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:06.416002 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:06.416033 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:06.416716 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:06.417825 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:06.417868 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:06.418272 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:06.419048 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:06.419261 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:06.419678 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:06.419689 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:06.419790 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:06.420188 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:06.420210 [ 32160 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:06.420663 [ 32160 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:06.420685 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:06.420955 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:06.421094 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:06.421238 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:06.421837 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:06.422068 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:06.422233 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:06.423049 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:06.423129 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:06.425031 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:06.425049 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:06.425441 [ 32068 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:06.425846 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:06.425891 [ 32067 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:06.426212 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:06.426249 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:06.426554 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:06.427703 [ 32075 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:06.427809 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:06.427956 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:06.428242 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:06.428296 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:06.430088 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:06.430204 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:06.430413 [ 32075 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1185ms, ratio: 0.025641028, points per minute: 12.982799)
2026.05.06 08:50:06.430589 [ 31904 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:06.430600 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:06.430752 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:06.430927 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:06.430984 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:06.431039 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:06.431144 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:06.431265 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:06.431359 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:06.431452 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:06.431563 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:06.432068 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:06.433205 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:06.433424 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:06.434075 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:06.434140 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:06.434253 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:06.434567 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:06.434866 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:06.434971 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:06.435245 [ 31917 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:06.435552 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:06.435851 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:06.436223 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:06.436356 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:06.436466 [ 32075 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:06.436503 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:06.436674 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:06.437339 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:06.439184 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:06.439646 [ 31887 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:06.440043 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:06.440421 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:06.440898 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:06.442335 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:06.442343 [ 32075 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:06.442463 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:06.442609 [ 32075 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:06.443427 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:06.443506 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:06.443680 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:06.443799 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:06.444331 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:06.444882 [ 32075 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:50:06.445192 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:06.445303 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:06.445408 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:06.447043 [ 32075 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:06.448221 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:06.449082 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:06.449308 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:06.449851 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:06.450373 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:06.450498 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:06.450658 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:06.451370 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:06.451832 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:06.454758 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:06.456841 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:06.456922 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:06.456997 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:06.457017 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:06.457120 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:06.457185 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:06.457236 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:06.457495 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.457739 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.457902 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:06.457999 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:06.458220 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:06.459198 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:06.459326 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:06.459468 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:06.459779 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.459986 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.460140 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:06.460233 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:06.460425 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:06.461626 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:06.461845 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:06.462041 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:06.462152 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:06.462339 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:06.462395 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.462655 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.462850 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:06.462897 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:06.462953 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:06.463147 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:06.463345 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:06.464253 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:06.464355 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:06.464405 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:06.464572 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:06.464768 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:06.464870 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.465087 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.465243 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:06.465337 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:06.465542 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:06.466669 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:06.466864 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:06.467059 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:06.467413 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.467756 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.467948 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:06.468068 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:06.468273 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:06.469305 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:06.469329 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:06.469433 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:06.469490 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:06.469600 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:06.469663 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:06.469869 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.469951 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.470070 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.470170 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.470224 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:06.470335 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:06.470334 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:06.470446 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:06.470672 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:06.470672 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:06.471674 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:06.471756 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:06.471824 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:06.471893 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:06.471970 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:06.472037 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:06.472232 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.472301 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.472437 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.472503 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:06.472611 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:06.472673 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:06.472706 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:06.472766 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:06.472908 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:06.472970 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:06.473510 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:06.474039 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:06.474183 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:06.474241 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:06.474327 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:06.474601 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.474674 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:06.474777 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:06.474811 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.474970 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:06.475067 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:06.475259 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:06.476314 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:06.476443 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:06.476598 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:06.476866 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.477059 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.477210 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:06.477303 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:06.477490 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:06.478580 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:06.478727 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:06.478898 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:06.479158 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.479219 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:06.479364 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.479539 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:06.479669 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:06.479867 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:06.480721 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:06.480861 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:06.480989 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:06.481129 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:06.481405 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.481431 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:06.481620 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.481773 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:06.481861 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:06.481879 [ 32073 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:06.482053 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:06.482798 [ 32273 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:06.482925 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:06.483059 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:06.483303 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.483497 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32273, kind: Regular, unbound: 0
2026.05.06 08:50:06.483688 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:06.483781 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:06.483968 [ 32273 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:06.484480 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:06.485166 [ 32273 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:06.485608 [ 32273 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:06.485699 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:06.489429 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:06.490924 [ 32273 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:06.491677 [ 32273 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:06.492100 [ 32133 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:06.492326 [ 32133 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:06.678644 [ 32156 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:06.680376 [ 32419 ] {} <Fatal> : Logical error: 'Ephemeral node /test/test_13/rdb/replicas/s9|r9/active still exists after 90s, probably it's owned by someone else. Either session_timeout_ms in client's config is different from server's config or it's a bug. Node data: '''.
2026.05.06 08:50:06.685514 [ 32156 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:06.695047 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:06.696136 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:06.719408 [ 32386 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:06.719536 [ 32386 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:06.719770 [ 32386 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006640848 sec. (150.583 rows/sec., 3.53 KiB/sec.)
2026.05.06 08:50:06.719836 [ 32386 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:06.723717 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:06.724348 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:06.731249 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:06.731961 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:06.743700 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:06.746728 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:06.747189 [ 32276 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:06.845570 [ 32419 ] {} <Fatal> : Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: DB::Exception::Exception<String const&, int, String&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<int>::type, std::type_identity<String&>::type>, String const&, int&&, String&) @ 0x000000003cd765b7
4. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:1071: zkutil::ZooKeeper::deleteEphemeralNodeIfContentMatches(String const&, String const&) @ 0x000000003cd4d4fa
5. ./build_docker/./src/Databases/DatabaseReplicatedWorker.cpp:0: DB::DatabaseReplicatedDDLWorker::initializeReplication() @ 0x0000000032bed841
6. ./build_docker/./src/Databases/DatabaseReplicatedWorker.cpp:0: DB::DatabaseReplicatedDDLWorker::initializeMainThread() @ 0x0000000032beab90
7. ./build_docker/./src/Interpreters/DDLWorker.cpp:1156: DB::DDLWorker::runMainThread() @ 0x0000000033a67db3
8. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x0000000033ab7ae6
9. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x0000000033ab789f
10. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
12. ? @ 0x00007f6e2fbe4ac3
13. ? @ 0x00007f6e2fc76850

2026.05.06 08:50:14.356002 [ 32171 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:14.356808 [ 32159 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:14.356798 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:14.357038 [ 32031 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:14.357393 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:14.357597 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:14.357774 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:14.358132 [ 31634 ] {} <Trace> BaseDaemon: Received signal 6
2026.05.06 08:50:14.358445 [ 31640 ] {} <Information> KeeperTCPHandler: Receiving request for session 5 took 7577 ms
2026.05.06 08:50:14.358761 [ 32555 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
2026.05.06 08:50:14.359078 [ 32555 ] {} <Fatal> BaseDaemon: (version 24.8.14.10546.altinitytest (altinity build), build id: DA078E6F75C2DC04F72E549EE0FB8F2D93941D6A, git hash: bd28307cd2efb6638c150abd8a7331dbf458804a) (from thread 32419) Received signal 6
2026.05.06 08:50:14.359323 [ 32555 ] {} <Fatal> BaseDaemon: Signal description: Aborted
2026.05.06 08:50:14.359467 [ 32555 ] {} <Fatal> BaseDaemon: 
2026.05.06 08:50:14.359706 [ 32555 ] {} <Fatal> BaseDaemon: Stack trace: 0x000055a36a4dc71e 0x000055a36ae023bf 0x000055a352759934 0x00007f6e2fb92520 0x00007f6e2fbe69fd 0x00007f6e2fb92476 0x00007f6e2fb787f3 0x000055a36a434730 0x000055a36a436a7f 0x000055a35277eb9d 0x000055a3871735b7 0x000055a38714a4fa 0x000055a37cfea841 0x000055a37cfe7b90 0x000055a37de64db3 0x000055a37deb4ae6 0x000055a37deb489f 0x000055a36a6d45dd 0x000055a36a6e6b24 0x00007f6e2fbe4ac3 0x00007f6e2fc76850
2026.05.06 08:50:14.359897 [ 32555 ] {} <Fatal> BaseDaemon: ########################################
2026.05.06 08:50:14.360159 [ 32555 ] {} <Fatal> BaseDaemon: (version 24.8.14.10546.altinitytest (altinity build), build id: DA078E6F75C2DC04F72E549EE0FB8F2D93941D6A, git hash: bd28307cd2efb6638c150abd8a7331dbf458804a) (from thread 32419) (no query) Received signal Aborted (6)
2026.05.06 08:50:14.360434 [ 32555 ] {} <Fatal> BaseDaemon: 
2026.05.06 08:50:14.360572 [ 32555 ] {} <Fatal> BaseDaemon: Stack trace: 0x000055a36a4dc71e 0x000055a36ae023bf 0x000055a352759934 0x00007f6e2fb92520 0x00007f6e2fbe69fd 0x00007f6e2fb92476 0x00007f6e2fb787f3 0x000055a36a434730 0x000055a36a436a7f 0x000055a35277eb9d 0x000055a3871735b7 0x000055a38714a4fa 0x000055a37cfea841 0x000055a37cfe7b90 0x000055a37de64db3 0x000055a37deb4ae6 0x000055a37deb489f 0x000055a36a6d45dd 0x000055a36a6e6b24 0x00007f6e2fbe4ac3 0x00007f6e2fc76850
2026.05.06 08:50:14.361704 [ 32186 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 14492ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:14.361799 [ 32129 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 8012ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:14.361940 [ 31826 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 9721ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:14.362022 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:14.362149 [ 32169 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 8003ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:14.362203 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:14.362332 [ 32126 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 15526ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:14.362426 [ 31791 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 10606ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:14.362470 [ 31768 ] {} <Trace> test_15.t (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 63742ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:14.362604 [ 31806 ] {} <Trace> test_15.t_r_ok (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 65019ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:14.362798 [ 32117 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 64304ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:14.362816 [ 31717 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 12994ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:14.363838 [ 32159 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:14.363920 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:14.364037 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:14.364132 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:14.364239 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:14.364569 [ 32051 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:14.364919 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:14.364942 [ 32051 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:14.364920 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:14.365035 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:14.365138 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:14.365202 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:14.365221 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:14.365541 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:14.365691 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:14.367048 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:14.367091 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:14.367263 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:14.367735 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:14.367927 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:14.368235 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:14.368785 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:14.369646 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:14.369747 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:14.370025 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:14.370028 [ 32052 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:14.370183 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:14.370296 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:14.370484 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:14.370496 [ 32052 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:14.370811 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:14.370838 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:14.371111 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:14.372809 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:14.373033 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:14.374392 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:14.374636 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:14.374982 [ 32047 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:14.375328 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:14.375338 [ 32047 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:14.375388 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:14.375456 [ 32159 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:14.375597 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:14.375726 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:14.375783 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:14.375818 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:14.376071 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:14.376586 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:14.377493 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:14.377745 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:14.378147 [ 32159 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 7931ms, ratio: 0.025641028, points per minute: 1.9398079)
2026.05.06 08:50:14.378344 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:14.378445 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:14.379004 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:14.379118 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:14.379225 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:14.379241 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:14.379322 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:14.379868 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:14.380189 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:14.380317 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:14.380420 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:14.381283 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:14.381943 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:14.382128 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:14.382424 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:14.382639 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:14.382897 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:14.384037 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:14.384185 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:14.384296 [ 32159 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:14.385043 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:14.385141 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:14.387307 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:14.390606 [ 32159 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:14.390728 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:14.390873 [ 32159 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:14.391761 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:14.391837 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:14.391955 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:14.392082 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:14.393702 [ 32159 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:50:14.395789 [ 32159 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 2ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:14.396394 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=7, jobs=7). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:14.396740 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=7, jobs=7). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202f17d4
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202e1d09
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202e158a
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::spawnThreads() @ 0x000000003afe272c
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0917
8. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003afe2f18
9. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
10. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
11. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
12. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
13. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
14. ? @ 0x00007f6e2fbe4ac3
15. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:14.397762 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:14.397943 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:14.398356 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:14.398906 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:14.399659 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:14.400018 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:14.404625 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:14.404818 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:14.405007 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:14.405323 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.405561 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.405727 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:14.405826 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:14.406036 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:14.406990 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:14.407120 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:14.407263 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:14.407521 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.407807 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.407993 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:14.408087 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:14.408281 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:14.409369 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:14.409496 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:14.409657 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:14.409910 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.410107 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.410263 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:14.410358 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:14.410563 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:14.411520 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:14.411713 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:14.411860 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:14.412114 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.412323 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.412477 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:14.412585 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:14.412776 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:14.413851 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:14.413978 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:14.414120 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:14.414374 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.414590 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.414746 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:14.414842 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:14.415039 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:14.416061 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:14.416191 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:14.416347 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:14.416616 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.416816 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.416968 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:14.417060 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:14.417253 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:14.418006 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:14.418126 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:14.418261 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:14.418509 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.418728 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:14.418874 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:14.418965 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:14.419156 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:14.419687 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:14.420426 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:14.420778 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:14.420854 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:14.424317 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:14.425385 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:14.425924 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:14.426288 [ 32041 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:14.426584 [ 32041 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:14.426910 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:14.426938 [ 32041 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:14.427141 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:14.429114 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:14.430335 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:14.431072 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:14.431142 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:14.431205 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:14.434010 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:14.436353 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:14.436507 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:14.440287 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:14.441417 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:14.441464 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:14.441556 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:14.441630 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:14.445149 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:14.445268 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:14.445633 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:14.446062 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:14.446864 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:14.447120 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:14.449726 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:14.449825 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:14.449921 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:14.450092 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.450221 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.450320 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:14.450382 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:14.450503 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:14.451103 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:14.451195 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:14.451287 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:14.451449 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.451588 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.451715 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:14.451778 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:14.451906 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:14.452646 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:14.452734 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:14.452830 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:14.452990 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.453118 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.453221 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:14.453284 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:14.453406 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:14.454072 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:14.454157 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:14.454248 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:14.454401 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.454543 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.454643 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:14.454705 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:14.454824 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:14.455513 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:14.455625 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:14.455722 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:14.455882 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.456012 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.456112 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:14.456177 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:14.456303 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:14.456963 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:14.457047 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:14.457150 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:14.457311 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.457434 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.457542 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:14.457601 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:14.457719 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:14.458201 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:14.458279 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:14.458366 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:14.458520 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.458653 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:14.458745 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:14.458802 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:14.458916 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:14.459294 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:14.459820 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:14.460101 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:14.460167 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:14.463152 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:14.463895 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:14.464199 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:14.464542 [ 32037 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:14.615429 [ 32555 ] {} <Fatal> BaseDaemon: 0.0. inlined from ./build_docker/./src/Common/StackTrace.cpp:349: StackTrace::tryCapture()
2026.05.06 08:50:14.615614 [ 32555 ] {} <Fatal> BaseDaemon: 0. ./build_docker/./src/Common/StackTrace.cpp:318: StackTrace::StackTrace(ucontext_t const&) @ 0x00000000200df71e
2026.05.06 08:50:15.025463 [ 32555 ] {} <Fatal> BaseDaemon: 1. ./build_docker/./src/Common/SignalHandlers.cpp:0: signalHandler(int, siginfo_t*, void*) @ 0x0000000020a053bf
2026.05.06 08:50:15.096587 [ 31640 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #5
2026.05.06 08:50:15.096663 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:50:15.097097 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:15.149250 [ 32040 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:15.155082 [ 32040 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:15.165400 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:15.166613 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:15.191677 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=784, jobs=664). (CANNOT_SCHEDULE_TASK) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:15.192220 [ 32281 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=784, jobs=664). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000202eb394
4. ./contrib/llvm-project/libcxx/include/string:1499: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000202d9869
5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000202db514
6. ./src/Common/ThreadPool.h:0: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000003b01a5ac
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000003b01bc2b
8. ./build_docker/./src/Interpreters/MutationsInterpreter.cpp:213: DB::isStorageTouchedByMutations(std::shared_ptr<DB::IMergeTreeDataPart const>, std::shared_ptr<DB::MergeTreeData::IMutationsSnapshot const>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::vector<DB::MutationCommand, std::allocator<DB::MutationCommand>> const&, std::shared_ptr<DB::Context const>) @ 0x00000000350a3d7a
9. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:2185: DB::MutateTask::prepare() @ 0x0000000039fc4116
10. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:2054: DB::MutateTask::execute() @ 0x0000000039fc1b76
11. ./build_docker/./src/Storages/MergeTree/MutatePlainMergeTreeTask.cpp:0: DB::MutatePlainMergeTreeTask::executeStep() @ 0x000000003a5099dc
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
13. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
15. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
16. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
17. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
18. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
19. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
20. ? @ 0x00007f6e2fbe4ac3
21. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:15.281172 [ 32029 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 66050ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:15.428128 [ 32555 ] {} <Fatal> BaseDaemon: 2. SignalAction(int, void*, void*) @ 0x000000000835c934
2026.05.06 08:50:15.428229 [ 32555 ] {} <Fatal> BaseDaemon: 3. ? @ 0x00007f6e2fb92520
2026.05.06 08:50:15.428267 [ 32555 ] {} <Fatal> BaseDaemon: 4. ? @ 0x00007f6e2fbe69fd
2026.05.06 08:50:15.428296 [ 32555 ] {} <Fatal> BaseDaemon: 5. ? @ 0x00007f6e2fb92476
2026.05.06 08:50:15.428323 [ 32555 ] {} <Fatal> BaseDaemon: 6. ? @ 0x00007f6e2fb787f3
2026.05.06 08:50:15.513216 [ 32020 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:15.513675 [ 32020 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:15.513721 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:15.514012 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:15.514120 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:15.514295 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:15.516840 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:15.516913 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:15.519070 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:15.519129 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:15.519309 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:15.519439 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:15.519574 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:15.519699 [ 32039 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:15.520126 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:15.520194 [ 32039 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:15.520314 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:15.520418 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:15.520442 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:15.520559 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:15.520572 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:15.520940 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:15.521243 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:15.522544 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:15.523187 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:15.523272 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:15.524238 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:15.524506 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:15.524921 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:15.525212 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:15.525470 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:15.525751 [ 32036 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:15.526360 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:15.526624 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:15.526850 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:15.527101 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:15.527153 [ 32017 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:15.527289 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:15.527421 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:15.527647 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:15.528023 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:15.531872 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:15.532191 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:15.533344 [ 32046 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:15.533447 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:15.533626 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:15.533687 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:15.533856 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:15.534636 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:15.534722 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:15.534795 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:15.535814 [ 32046 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1141ms, ratio: 0.025641028, points per minute: 13.48345)
2026.05.06 08:50:15.536043 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:15.536142 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:15.536228 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:15.537005 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:15.537108 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:15.537206 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:15.537319 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:15.537415 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:15.537723 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:15.539337 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:15.539510 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:15.540684 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:15.540924 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:15.541323 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:15.541693 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:15.541770 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:15.541802 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:15.541890 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:15.542005 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:15.543198 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:15.543340 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:15.543471 [ 32046 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:15.546718 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:15.546898 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:15.547436 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:15.547964 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:15.548819 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:15.548907 [ 32046 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:15.549004 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:15.549115 [ 32046 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:15.549172 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:15.551880 [ 32046 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:50:15.553704 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:15.553845 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:15.553992 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:15.554268 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.554496 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.554672 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:15.554767 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:15.554973 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:15.555950 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:15.556080 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:15.556222 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:15.556473 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.556688 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.556840 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:15.556931 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:15.557121 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:15.558188 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:15.558310 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:15.558448 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:15.558712 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.558907 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.559060 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:15.559167 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:15.559354 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:15.560350 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:15.560474 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:15.560627 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:15.560873 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.561066 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.561216 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:15.561308 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:15.561493 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:15.562560 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:15.562685 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:15.562825 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:15.563083 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.563282 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.563447 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:15.563557 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:15.563783 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:15.564742 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:15.564863 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:15.565003 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:15.565252 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.565446 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.565611 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:15.565699 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:15.565886 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:15.566623 [ 32278 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:15.566739 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:15.566874 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:15.567116 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.567322 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32278, kind: Regular, unbound: 0
2026.05.06 08:50:15.567464 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:15.567571 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:15.567809 [ 32278 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:15.568321 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:15.569010 [ 32278 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:15.569400 [ 32278 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:15.569489 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:15.573965 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:15.575286 [ 32278 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:15.576079 [ 32278 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:15.576648 [ 32022 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:15.577063 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:15.577104 [ 32022 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:15.577461 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:15.579499 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:15.581232 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:15.581993 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:15.582100 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:15.582200 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:15.584956 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:15.587254 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:15.587409 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:15.591426 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:15.593787 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:15.593864 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:15.593978 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:15.594083 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:15.598274 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:15.598450 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:15.598982 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:15.599534 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:15.600466 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:15.600838 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:15.605315 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:15.605456 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:15.605618 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:15.605882 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.606090 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.606244 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:15.606338 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:15.606552 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:15.607439 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:15.607583 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:15.607755 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:15.608005 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.608203 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.608353 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:15.608443 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:15.608646 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:15.609688 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:15.609811 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:15.609947 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:15.610189 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.610378 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.610544 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:15.610636 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:15.610820 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:15.611836 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:15.611966 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:15.612104 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:15.612349 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.612558 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.612709 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:15.612801 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:15.612985 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:15.614038 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:15.614160 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:15.614297 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:15.614560 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.614758 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.614925 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:15.615019 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:15.615210 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:15.616237 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:15.616364 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:15.616504 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:15.616768 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.616960 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.617108 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:15.617196 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:15.617383 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:15.618123 [ 32279 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:15.618240 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:15.618373 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:15.618630 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.618823 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32279, kind: Regular, unbound: 0
2026.05.06 08:50:15.618982 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:15.619071 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:15.619256 [ 32279 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:15.619836 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:15.620515 [ 32279 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:15.620924 [ 32279 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:15.621032 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:15.625097 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:15.626556 [ 32279 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:15.627227 [ 32279 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:15.627646 [ 32030 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:15.627821 [ 32030 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:15.801848 [ 32001 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 62199ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:15.914993 [ 32555 ] {} <Fatal> BaseDaemon: 7.0. inlined from ./contrib/llvm-project/libcxx/include/atomic:958: int std::__cxx_atomic_load[abi:v15007]<int>(std::__cxx_atomic_base_impl<int> const*, std::memory_order)
2026.05.06 08:50:15.915101 [ 32555 ] {} <Fatal> BaseDaemon: 7.1. inlined from ./contrib/llvm-project/libcxx/include/atomic:1560: std::__atomic_base<int, false>::load[abi:v15007](std::memory_order) const
2026.05.06 08:50:15.915147 [ 32555 ] {} <Fatal> BaseDaemon: 7.2. inlined from ./contrib/llvm-project/libcxx/include/atomic:1564: std::__atomic_base<int, false>::operator int[abi:v15007]() const
2026.05.06 08:50:15.915197 [ 32555 ] {} <Fatal> BaseDaemon: 7.3. inlined from ./base/poco/Foundation/include/Poco/Logger.h:2354: Poco::Logger::is(int) const
2026.05.06 08:50:15.915241 [ 32555 ] {} <Fatal> BaseDaemon: 7. ./build_docker/./src/Common/Exception.cpp:47: DB::abortOnFailedAssertion(String const&, void* const*, unsigned long, unsigned long) @ 0x0000000020037730
2026.05.06 08:50:16.010452 [ 32000 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.010912 [ 32000 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.010954 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:16.011299 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.011333 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:16.011841 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.013881 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.013935 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.015953 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.016078 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.016290 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:16.016498 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:16.016614 [ 31998 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.016652 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:16.016922 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:16.016959 [ 31998 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.017297 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.017319 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:16.017392 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:16.017498 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:16.017618 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.017633 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:16.018268 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:16.019028 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.019241 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.019800 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:16.020506 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:16.020754 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.020784 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:16.021077 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.021132 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:16.021185 [ 32021 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.021604 [ 32021 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.021617 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:16.021683 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:16.021858 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:16.021893 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.022110 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.023055 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:16.023204 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:16.023322 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:16.023764 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.023816 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.025554 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.025612 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.025901 [ 32025 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.026282 [ 32025 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.026279 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:16.026564 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:16.026683 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.026832 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.028704 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.028718 [ 31999 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:16.028743 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.028830 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:16.028960 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:16.030377 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:16.030490 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.031135 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:16.031244 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:16.031348 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:16.031417 [ 31999 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 479ms, ratio: 0.025641028, points per minute: 32.118195)
2026.05.06 08:50:16.031640 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:16.031740 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:16.032501 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:16.032634 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:16.032756 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:16.033248 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:16.034221 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:16.034826 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:16.035782 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:16.036004 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:16.036350 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:16.036550 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:16.036703 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:16.036839 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:16.038149 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:16.038285 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:16.038396 [ 31999 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:16.040745 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:16.043049 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:16.043139 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:16.043273 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:16.043420 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:16.043887 [ 31999 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:16.044006 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:16.044129 [ 31999 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:16.046959 [ 31999 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 16ms, ratio: 0.025641028, points per minute: 961.5385)
2026.05.06 08:50:16.048721 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:16.048935 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:16.049413 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:16.050084 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:16.051093 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:16.051494 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:16.057085 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:16.057242 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:16.057400 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:16.057715 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.057966 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.058138 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:16.058241 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:16.058456 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:16.059422 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:16.059573 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:16.059752 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:16.060023 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.060246 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.060405 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:16.060501 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:16.060715 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:16.061823 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:16.061986 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:16.062148 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:16.062412 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.062639 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.062823 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:16.062924 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:16.063120 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:16.064254 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:16.064393 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:16.064558 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:16.064837 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.065041 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.065199 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:16.065297 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:16.065493 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:16.066663 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:16.066803 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:16.066976 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:16.067267 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.067486 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.067733 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:16.067846 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:16.068082 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:16.069257 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:16.069394 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:16.069567 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:16.069833 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.070040 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.070227 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:16.070327 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:16.070543 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:16.071456 [ 32280 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:16.071645 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:16.071799 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:16.072070 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.072276 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32280, kind: Regular, unbound: 0
2026.05.06 08:50:16.072433 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:16.072570 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:16.072774 [ 32280 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:16.072806 [ 31985 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:16.073416 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:16.074295 [ 32280 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:16.074764 [ 32280 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:16.074891 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:16.076404 [ 31985 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:16.079250 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:16.080694 [ 32280 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:16.081367 [ 32280 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:16.081897 [ 31983 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.082290 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:16.082329 [ 31983 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:16.082716 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.084761 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.085620 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:16.086415 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:16.086687 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:16.087177 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:16.087286 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:16.087386 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:16.090102 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:16.092395 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:16.092566 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:16.096543 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:16.099003 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:16.099087 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:16.099241 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:16.099418 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:16.104357 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:16.104561 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:16.105126 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:16.105741 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:16.106800 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:16.107320 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:16.109506 [ 32436 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:16.109685 [ 32436 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:16.110030 [ 32436 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006971319 sec. (143.445 rows/sec., 3.36 KiB/sec.)
2026.05.06 08:50:16.110124 [ 32436 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:16.112304 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:16.112506 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:16.112739 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:16.113052 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.113272 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.113433 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:16.113549 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:16.113752 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:16.114484 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:16.114722 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:16.114849 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:16.114988 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:16.115112 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:16.115244 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.115439 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.115641 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:16.115737 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:16.115929 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:16.116992 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:16.117115 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:16.117276 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:16.117641 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.117910 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.118129 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:16.118256 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:16.118465 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:16.119474 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:16.119650 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:16.119793 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:16.120044 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.120240 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.120391 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:16.120484 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:16.120684 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:16.121759 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:16.121804 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:16.121886 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:16.122024 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:16.122280 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.122441 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:16.122496 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.122696 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:16.122793 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:16.122988 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:16.124207 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:16.124382 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:16.124552 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:16.124816 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.125030 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.125185 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:16.125151 [ 32191 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1763ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:16.125274 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:16.125466 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:16.126233 [ 32275 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:16.126368 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:16.126505 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:16.126780 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.126971 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32275, kind: Regular, unbound: 0
2026.05.06 08:50:16.127112 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:16.127200 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:16.127388 [ 32275 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:16.127955 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:16.128824 [ 32275 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:16.129364 [ 32275 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:16.129497 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:16.132766 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:16.134152 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:16.134449 [ 32275 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:16.135334 [ 32275 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:16.135781 [ 31977 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:16.137148 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:16.137627 [ 32282 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:16.221572 [ 32056 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1860ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:16.349793 [ 31975 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:16.353654 [ 31975 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:16.363406 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:16.364553 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:16.392836 [ 32388 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:16.392997 [ 32388 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:16.393310 [ 32388 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.008799141 sec. (113.647 rows/sec., 2.66 KiB/sec.)
2026.05.06 08:50:16.393401 [ 32388 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:16.398212 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:16.398912 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:16.405913 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:16.406597 [ 32461 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:16.407206 [ 32555 ] {} <Fatal> BaseDaemon: 8.0. inlined from ./build_docker/./src/Common/Exception.cpp:208: DB::Exception::getStackFramePointers() const
2026.05.06 08:50:16.407307 [ 32555 ] {} <Fatal> BaseDaemon: 8. ./build_docker/./src/Common/Exception.cpp:115: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x0000000020039a7f
2026.05.06 08:50:16.418988 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:16.422569 [ 32274 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:16.423122 [ 32274 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:16.686943 [ 32555 ] {} <Fatal> BaseDaemon: 9. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
2026.05.06 08:50:16.815884 [ 31958 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 2454ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:16.974449 [ 32014 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.974907 [ 32014 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.974934 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:16.975182 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:16.975295 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.975472 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.977887 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.977954 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.979945 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.980012 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.980181 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:16.980299 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:16.980389 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:16.980518 [ 31949 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.980951 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:16.981004 [ 31949 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.981068 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:16.981190 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:16.981278 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.981301 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:16.981279 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:16.981624 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.981908 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:16.983043 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.983160 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:16.983165 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.983950 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:16.984211 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:16.984599 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:16.984619 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.984807 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.984946 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:16.985021 [ 31947 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.985574 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:16.985603 [ 31947 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.985862 [ 32282 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.985973 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:16.986313 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.986454 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:16.986667 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:16.986810 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:16.987493 [ 32282 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.987923 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.989049 [ 32282 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.989472 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.989836 [ 31940 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.990202 [ 31940 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.990200 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:16.990445 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:16.990557 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.990665 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.992478 [ 32286 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.992545 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.993744 [ 31953 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:16.993848 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:16.993997 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:16.994135 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:16.994176 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.994574 [ 32153 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.994947 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:16.995005 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:16.995118 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:16.995218 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:16.995224 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:16.996690 [ 31953 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 949ms, ratio: 0.025641028, points per minute: 16.211397)
2026.05.06 08:50:16.996933 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:16.996996 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:16.997066 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:16.997799 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:16.997924 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:16.997947 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:16.998108 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:16.998796 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:16.998965 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:16.999343 [ 31933 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:16.999711 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:16.999992 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:17.000206 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:17.000285 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:17.000350 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:17.000935 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:17.001120 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:17.001426 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:17.001605 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:17.001866 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:17.003054 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:17.003189 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:17.003298 [ 31953 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:17.003824 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:17.004172 [ 31937 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:17.004453 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:17.004544 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:17.004873 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:17.006764 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:17.006837 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:17.006954 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:17.007065 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:17.007090 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:17.008610 [ 31953 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:17.008710 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:17.008829 [ 31953 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:17.008916 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:17.009775 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:17.009912 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:17.010049 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:17.011354 [ 31953 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:50:17.011686 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:17.011866 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:17.012345 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:17.012981 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:17.012984 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:17.014080 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:17.014436 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:17.014895 [ 31953 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 4ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:17.015519 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:17.015713 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:17.019015 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:17.019156 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:17.019316 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:17.019660 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:17.019652 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.019939 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.020132 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:17.020273 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:17.020488 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:17.021552 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:17.021686 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:17.021832 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:17.022139 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.022311 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:17.022359 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.022389 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:17.022550 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:17.022569 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:17.022650 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:17.022699 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:17.022847 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:17.024025 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:17.024170 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:17.024314 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:17.024582 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.024781 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.024945 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:17.025052 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:17.025241 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:17.026255 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:17.026396 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:17.026554 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:17.026808 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.027003 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.027159 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:17.027254 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:17.027443 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:17.027749 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:17.027967 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:17.028595 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:17.028649 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:17.028733 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:17.028877 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:17.029136 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.029222 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:17.029358 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.029517 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:17.029630 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:17.029827 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:17.030202 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:17.030633 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:17.031019 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:17.031147 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:17.031293 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:17.031569 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.031806 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.031964 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:17.032055 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:17.032250 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:17.033061 [ 32286 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:17.033184 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:17.033322 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:17.033588 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.033786 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32286, kind: Regular, unbound: 0
2026.05.06 08:50:17.033934 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:17.034024 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:17.034214 [ 32286 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:17.034788 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:17.035556 [ 32286 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:17.035994 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:17.036072 [ 32286 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:17.036149 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:17.036171 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:17.036331 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:17.036669 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.036927 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.037100 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:17.037198 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:17.037457 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:17.038477 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:17.038632 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:17.038782 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:17.039050 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.039264 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.039467 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:17.039591 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:17.039821 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:17.039972 [ 32286 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:17.040964 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:17.041110 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:17.041264 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:17.041296 [ 32286 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:17.041586 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.041814 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.041989 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:17.041938 [ 32286 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:17.042085 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:17.042278 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:17.043360 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:17.043546 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:17.043734 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:17.044016 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.044253 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.044413 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:17.044508 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:17.044715 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:17.045981 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:17.046111 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:17.046274 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:17.046592 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.046820 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.046978 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:17.047072 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:17.047307 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:17.048402 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:17.048556 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:17.048700 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:17.048976 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.049171 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.049386 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:17.049487 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:17.049699 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:17.050519 [ 32277 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:17.050665 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:17.050821 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:17.051074 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.051317 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32277, kind: Regular, unbound: 0
2026.05.06 08:50:17.051496 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:17.051666 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:17.051863 [ 32277 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:17.052446 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:17.053145 [ 32277 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:17.053636 [ 32277 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:17.053734 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:17.058074 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:17.059729 [ 32277 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:17.060514 [ 32277 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:17.060822 [ 31931 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:17.291475 [ 31966 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:17.297582 [ 31966 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:17.306795 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:17.307885 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:17.329321 [ 32347 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:17.329427 [ 32347 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:17.329777 [ 32347 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006003368 sec. (166.573 rows/sec., 3.90 KiB/sec.)
2026.05.06 08:50:17.329865 [ 32347 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:17.334135 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:17.334770 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:17.341744 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:17.342469 [ 32391 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:17.354429 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:17.357463 [ 32276 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:17.357931 [ 32276 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:17.520867 [ 31922 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1396ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:17.538440 [ 32318 ] {} <Information> AWSClient: Response status: 404, Not Found
2026.05.06 08:50:17.538990 [ 32318 ] {} <Information> AWSClient: AWSXmlClient: HTTP response code: 404
Resolved remote host IP address: 127.0.0.1:11111
Request ID: 
Exception name: 
Error message: No response body.
13 response headers:
accept-ranges : bytes
content-length : 0
content-type : 
date : Wed, 06 May 2026 06:50:17 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-id-2 : 7dc7eb22d3288ec80374614e9088e31d3668a6922ead55932dd2a8e56373820f
x-amz-request-id : 18ACE7409921B679
x-content-type-options : nosniff
x-minio-error-code : NoSuchKey
x-minio-error-desc : "The specified key does not exist."
x-xss-protection : 1; mode=block
2026.05.06 08:50:17.539400 [ 32318 ] {} <Information> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2026.05.06 08:50:17.714510 [ 32555 ] {} <Fatal> BaseDaemon: 10. ./src/Common/Exception.h:0: DB::Exception::Exception<String const&, int, String&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<int>::type, std::type_identity<String&>::type>, String const&, int&&, String&) @ 0x000000003cd765b7
2026.05.06 08:50:18.032485 [ 31919 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.033039 [ 31919 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.033143 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:18.033428 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.033579 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.033740 [ 32274 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.036064 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.036189 [ 32274 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.038031 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.038075 [ 32274 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.038321 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:18.038472 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:18.038601 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:18.038641 [ 31916 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.039034 [ 31916 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.039037 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:18.039276 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.039286 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:18.039345 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.039416 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:18.039553 [ 32272 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.039570 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:18.040221 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:18.041220 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.041347 [ 32272 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.041840 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:18.042562 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:18.042748 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:18.043019 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.043059 [ 32272 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.043144 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:18.043435 [ 31911 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.043731 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:18.043844 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:18.043919 [ 31911 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.044200 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.044231 [ 32273 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.044440 [ 32275 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.045125 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:18.045271 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:18.045398 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:18.046019 [ 32273 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.046092 [ 32275 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.047795 [ 32273 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.047878 [ 32275 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.048185 [ 31912 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.048690 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:18.048713 [ 31912 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.049027 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.049082 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.049353 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.050481 [ 31920 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:18.050600 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:18.050754 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:18.050843 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.051076 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.052510 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:18.052784 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.053159 [ 31920 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 1038ms, ratio: 0.025641028, points per minute: 14.8214035)
2026.05.06 08:50:18.053326 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:18.053331 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:18.053464 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:18.053470 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:18.053595 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:18.054074 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:18.054158 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:18.054244 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:18.054720 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:18.056192 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:18.056380 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:18.057011 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:18.057177 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:18.057504 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:18.057942 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:18.058712 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:18.058859 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:18.059266 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:18.059383 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:18.059484 [ 31920 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:18.062802 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:18.064644 [ 31920 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:18.064755 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:18.064864 [ 31920 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:18.065232 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:18.065305 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:18.065423 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:18.065552 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:18.067591 [ 31920 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 14ms, ratio: 0.025641028, points per minute: 1098.9011)
2026.05.06 08:50:18.070409 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:18.070603 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:18.071185 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:18.071796 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:18.072703 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:18.073068 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:18.077595 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:18.077738 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:18.077886 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:18.078176 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.078387 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.078561 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:18.078657 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:18.078875 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:18.079842 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:18.079972 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:18.080112 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:18.080369 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.080584 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.080737 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:18.080829 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:18.081018 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:18.082078 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:18.082203 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:18.082340 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:18.082604 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.082800 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.082970 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:18.083066 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:18.083255 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:18.084259 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:18.084385 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:18.084523 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:18.084790 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.084983 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.085131 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:18.085223 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:18.085409 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:18.086462 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:18.086606 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:18.086746 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:18.087000 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.087215 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.087365 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:18.087458 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:18.087695 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:18.088672 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:18.088796 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:18.088935 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:18.089184 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.089375 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.089522 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:18.089630 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:18.089818 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:18.090561 [ 32284 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:18.090679 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:18.090812 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:18.091071 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.091264 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32284, kind: Regular, unbound: 0
2026.05.06 08:50:18.091406 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:18.091495 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:18.091755 [ 32284 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:18.092288 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:18.092970 [ 32284 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:18.093362 [ 32284 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:18.093451 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:18.098228 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:18.099548 [ 32284 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.100228 [ 32284 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.100659 [ 31902 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:18.100896 [ 31902 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.101232 [ 31902 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:18.101305 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.101646 [ 32283 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.103955 [ 32283 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.106033 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:18.106855 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:18.106956 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:18.107051 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:18.110228 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:18.112556 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:18.112696 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:18.116658 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:18.118979 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:18.119051 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:18.119164 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:18.119272 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:18.123894 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:18.124075 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:18.124465 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:18.125055 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:18.125927 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:18.126276 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:18.126824 [ 31901 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 65301ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:18.130851 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:18.130993 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:18.131136 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:18.131393 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.131657 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.131815 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:18.131920 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:18.132115 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:18.133045 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:18.133170 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:18.133307 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:18.133571 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.133770 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.133920 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:18.134011 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:18.134198 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:18.135251 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:18.135376 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:18.135512 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:18.135816 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.136011 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.136179 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:18.136272 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:18.136458 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:18.137409 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:18.137551 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:18.137689 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:18.137934 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.138126 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.138274 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:18.138367 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:18.138565 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:18.139692 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:18.139821 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:18.139960 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:18.140213 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.140432 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.140597 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:18.140692 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:18.140883 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:18.141837 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:18.141962 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:18.142099 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:18.142345 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.142552 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.142697 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:18.142784 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:18.142971 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:18.143734 [ 32283 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:18.143855 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:18.144002 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:18.144244 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.144436 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32283, kind: Regular, unbound: 0
2026.05.06 08:50:18.144600 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:18.144689 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:18.144877 [ 32283 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:18.145380 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:18.146053 [ 32283 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:18.146443 [ 32283 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:18.146551 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:18.150358 [ 32283 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:18.151734 [ 32283 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.152388 [ 32283 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.152777 [ 31896 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:18.152985 [ 31896 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000003 of type MUTATE_PART for part all_1_1_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:18.187183 [ 32150 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:18.193387 [ 32150 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:18.202822 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:18.203976 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:18.225810 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:18.225945 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:18.226229 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006428648 sec. (155.554 rows/sec., 3.65 KiB/sec.)
2026.05.06 08:50:18.226312 [ 32460 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:18.230197 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:18.230867 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:18.237624 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:18.238340 [ 32393 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:18.249894 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.252795 [ 32280 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.253252 [ 32280 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:18.471345 [ 31882 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:18.477406 [ 31882 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:18.486611 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:18.487687 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:18.509993 [ 32435 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:18.510127 [ 32435 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:18.510413 [ 32435 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.007174669 sec. (139.379 rows/sec., 3.27 KiB/sec.)
2026.05.06 08:50:18.510496 [ 32435 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:18.514683 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:18.515320 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:18.522038 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:18.522764 [ 32362 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:18.535362 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.538286 [ 32273 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.538750 [ 32273 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:18.618592 [ 31880 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.618991 [ 31880 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.619081 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:18.619401 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.619421 [ 32284 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.619929 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.622705 [ 32284 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.622814 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.624458 [ 32284 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.624662 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.624684 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 parts to remove and reached the limit for one removal iteration
2026.05.06 08:50:18.624824 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_0_0_0_2]
2026.05.06 08:50:18.624930 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:18.625089 [ 31881 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.625406 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:18.625475 [ 31881 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.625706 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_0_0_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:18.625763 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.625785 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.625863 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:18.625989 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_0_0_0_2]
2026.05.06 08:50:18.626075 [ 32279 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.626642 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:18.627649 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.627845 [ 32279 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.628179 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_0_0_0 for part all_0_0_0_2
2026.05.06 08:50:18.628989 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_0_0_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:18.629184 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 for part all_0_0_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:18.629314 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.629616 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:18.629675 [ 32279 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.629676 [ 31869 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.630086 [ 31869 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.630090 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:18.630137 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:18.630347 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.630415 [ 32280 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.630630 [ 32277 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.631496 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys
2026.05.06 08:50:18.631688 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:18.631805 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_0_0_0_2 cannot be removed
2026.05.06 08:50:18.632112 [ 32280 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.632193 [ 32277 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.634027 [ 32280 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_0_0_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.634070 [ 32277 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.634377 [ 31860 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.634734 [ 31860 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.634787 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:18.635014 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.635181 [ 32271 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.635284 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.636996 [ 31877 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_0_0_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:18.637107 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_0_0_0_2]
2026.05.06 08:50:18.637238 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:18.637359 [ 32271 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutation of part all_0_0_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.637376 [ 32285 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.639134 [ 32285 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.639318 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_0_0_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:18.639503 [ 31857 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.639809 [ 31877 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 572ms, ratio: 0.025641028, points per minute: 26.896183)
2026.05.06 08:50:18.639972 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.640043 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 old parts to remove. Parts: [all_1_1_0_2]
2026.05.06 08:50:18.640147 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 old parts from ZooKeeper
2026.05.06 08:50:18.640247 [ 32276 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.640299 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:18.640416 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:18.640522 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:18.640711 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): There is no part all_1_1_0_2 in ZooKeeper, it was only in filesystem
2026.05.06 08:50:18.640798 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts from ZooKeeper. Removing them from filesystem.
2026.05.06 08:50:18.640886 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from filesystem (serially): Parts: [all_1_1_0_2]
2026.05.06 08:50:18.641360 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 local references is zero, will check blobs can be removed in zookeeper
2026.05.06 08:50:18.641920 [ 32276 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.642872 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found mutation parent all_1_1_0 for part all_1_1_0_2
2026.05.06 08:50:18.643362 [ 32526 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55086, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:18.643518 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found files not to remove from parent part all_1_1_0: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, columns.txt, n.cmrk2, k.bin]
2026.05.06 08:50:18.643582 [ 32276 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.643738 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing zookeeper lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 for part all_1_1_0_2 (files to keep: [count.txt, m.bin, n.bin, k.cmrk2, default_compression_codec.txt, metadata_version.txt, serialization.json, primary.cidx, m.cmrk2, k.bin, n.cmrk2, columns.txt])
2026.05.06 08:50:18.643998 [ 31855 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.644040 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 130
2026.05.06 08:50:18.644374 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.644475 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:18.644765 [ 32278 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.645718 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Found 1 (2) zookeeper locks for /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp
2026.05.06 08:50:18.645762 [ 32526 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_0_0_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:18.645862 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Blobs of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:18.645924 [ 32526 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_0_0_0_2
2026.05.06 08:50:18.645971 [ 31877 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Some blobs (columns.txt, n.cmrk2, k.bin, m.cmrk2, primary.cidx, serialization.json, metadata_version.txt, default_compression_codec.txt, k.cmrk2, n.bin, m.bin, count.txt) of part all_1_1_0_2 cannot be removed
2026.05.06 08:50:18.646605 [ 32278 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.648376 [ 32278 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Part all_1_1_0_2 should be deleted after previous attempt before fetch
2026.05.06 08:50:18.650107 [ 32526 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:18.651710 [ 31877 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): removePartsFinally: removing all_1_1_0_2 (state Deleting) from data_parts_indexes
2026.05.06 08:50:18.651816 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removing 1 parts from memory: Parts: [all_1_1_0_2]
2026.05.06 08:50:18.651933 [ 31877 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Removed 1 old parts
2026.05.06 08:50:18.652466 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:18.652561 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:18.652688 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:18.652805 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:18.654558 [ 31877 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0.25641027, interval: 15ms, ratio: 0.025641028, points per minute: 1025.6411)
2026.05.06 08:50:18.656828 [ 31877 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 3ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:18.657559 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk.
2026.05.06 08:50:18.657750 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1
2026.05.06 08:50:18.658151 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:18.658657 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:18.659489 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/s3_ryi_hxdgqzgklpzwmudbqytftnblikeys/1 created
2026.05.06 08:50:18.659920 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:18.664456 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/clj/yfyubsniuixdmrdrcjufulihucwys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/columns.txt)
2026.05.06 08:50:18.664641 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Reset state for seek to position 0
2026.05.06 08:50:18.664795 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:18.665099 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.665325 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: c19724c2f984163665bde94656daf920, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.665486 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:18.665599 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:18.665805 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c19724c2f984163665bde94656daf920): Key: c19724c2f984163665bde94656daf920. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:18.666837 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ynw/kptwbmdylnrwzhjuwdtvifbkdycfa (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/serialization.json)
2026.05.06 08:50:18.666976 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Reset state for seek to position 0
2026.05.06 08:50:18.667126 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:18.667397 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.667652 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 138075dc4f908b8696832089aba07a75, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.667817 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:18.667913 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:18.668116 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(138075dc4f908b8696832089aba07a75): Key: 138075dc4f908b8696832089aba07a75. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:18.669227 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/kit/ijcnjlxcuqkmbfkborlzhefefkuho (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/metadata_version.txt)
2026.05.06 08:50:18.669359 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Reset state for seek to position 0
2026.05.06 08:50:18.669499 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:18.669771 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.669970 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: c398d2bbee132ab3d078997de3f82f98, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.670127 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:18.670222 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:18.670416 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(c398d2bbee132ab3d078997de3f82f98): Key: c398d2bbee132ab3d078997de3f82f98. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:18.671431 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ryi/hxdgqzgklpzwmudbqytftnblikeys (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/checksums.txt)
2026.05.06 08:50:18.671576 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Reset state for seek to position 0
2026.05.06 08:50:18.671749 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:18.672009 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.672208 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: 996dedb7dc3c5a127eefbc86122b0cfc, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.672364 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:18.672473 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:18.672679 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(996dedb7dc3c5a127eefbc86122b0cfc): Key: 996dedb7dc3c5a127eefbc86122b0cfc. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:18.673769 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/suj/pdjtrmklydqzugwdlggofnonjlrvh (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/n.cmrk2)
2026.05.06 08:50:18.673900 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Reset state for seek to position 0
2026.05.06 08:50:18.674045 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:18.674329 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.674552 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 3244d0979a74840152a85a31e615952a, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.674709 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:18.674806 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:18.675006 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(3244d0979a74840152a85a31e615952a): Key: 3244d0979a74840152a85a31e615952a. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:18.676040 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/pxw/kxjaktghoepqudkmwjbltrnsnkdwu (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/count.txt)
2026.05.06 08:50:18.676169 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Reset state for seek to position 0
2026.05.06 08:50:18.676315 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:18.676583 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.676792 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: eb35a528d8f8ea53396c56179f8107bc, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.676976 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:18.677084 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:18.677278 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(eb35a528d8f8ea53396c56179f8107bc): Key: eb35a528d8f8ea53396c56179f8107bc. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:18.678065 [ 32271 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/nxv/ojcdzswtdwvcejlmfcelsavrzkzri (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_0_0_0_2/default_compression_codec.txt)
2026.05.06 08:50:18.678188 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Reset state for seek to position 0
2026.05.06 08:50:18.678324 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:18.678603 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.678819 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 786f9b3fa20c683a39db0316dde3086f, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32271, kind: Regular, unbound: 0
2026.05.06 08:50:18.678964 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:18.679068 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:18.679259 [ 32271 ] {} <Test> CachedOnDiskReadBufferFromFile(786f9b3fa20c683a39db0316dde3086f): Key: 786f9b3fa20c683a39db0316dde3086f. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:18.679895 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_0_0_0_2 unique id s3/ryi/hxdgqzgklpzwmudbqytftnblikeys metadata onto disk s3_disk finished.
2026.05.06 08:50:18.680601 [ 32271 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_0_0_0_2 to all_0_0_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:18.681009 [ 32271 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_0_0_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:18.681100 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_0_0_0_2 to zookeeper
2026.05.06 08:50:18.684540 [ 32271 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_0_0_0_2.
2026.05.06 08:50:18.685862 [ 32271 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.686491 [ 32271 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.686920 [ 32181 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:18.687174 [ 32181 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:18.687485 [ 32181 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:18.687509 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:18.687901 [ 32281 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:18.689795 [ 32281 ] {} <Debug> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutation of part all_1_1_0_2 finished by some other replica, will download mutated part
2026.05.06 08:50:18.691512 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Fetching part all_1_1_0_2 from default:/test/02485/test_13/rmt/replicas/2
2026.05.06 08:50:18.692345 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Trying to fetch with zero-copy replication, but disk is not provided, will try to select
2026.05.06 08:50:18.692461 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Checking disk s3_disk with type s3
2026.05.06 08:50:18.692587 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk s3_disk (with type s3) supports zero-copy replication
2026.05.06 08:50:18.695278 [ 31642 ] {} <Trace> InterserverIOHTTPHandler-factory: HTTP Request for InterserverIOHTTPHandler-factory. Method: POST, Address: [::ffff:172.17.0.2]:55070, User-Agent: ClickHouse/24.8.14.10546.altinitytest (altinity build), Length: 0, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2026.05.06 08:50:18.697624 [ 31642 ] {} <Trace> InterserverIOHTTPHandler: Request URI: /?endpoint=DataPartsExchange%3Adefault%3A%2Ftest%2F02485%2Ftest_13%2Frmt%2Freplicas%2F2&part=all_1_1_0_2&client_protocol_version=8&compress=false&remote_fs_metadata=s3
2026.05.06 08:50:18.697777 [ 31642 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768) (Replicated PartsService): Sending part all_1_1_0_2
2026.05.06 08:50:18.701666 [ 31642 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
2026.05.06 08:50:18.704164 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using storage balanced reservation
2026.05.06 08:50:18.704248 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, reserving space using TTL rules
2026.05.06 08:50:18.704376 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Trying to reserve 1.00 MiB on the selected disk: s3_disk (with type s3)
2026.05.06 08:50:18.704498 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Disk for fetch is not provided, getting disk from reservation s3_disk with type 's3'
2026.05.06 08:50:18.709233 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk.
2026.05.06 08:50:18.709424 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Set zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1
2026.05.06 08:50:18.709986 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 190
2026.05.06 08:50:18.710595 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:18.711631 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zookeeper temporary ephemeral lock /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/s3_jsj_xyihxmffegxddtcontnqlpnwidbrp/1 created
2026.05.06 08:50:18.712025 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Downloading files 13
2026.05.06 08:50:18.716540 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ips/rjktgzqkhukijlvlkawyplbhgtbqf (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/columns.txt)
2026.05.06 08:50:18.716708 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Reset state for seek to position 0
2026.05.06 08:50:18.716869 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Having 1 file segments to read: [0, 66], current read range: [0, 67)
2026.05.06 08:50:18.717167 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.717391 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 66], key: 4f35d3dc379904cf70c62240c71a85ac, state: DOWNLOADED, downloaded size: 67, reserved size: 67, downloader id: None, current write offset: 67, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.717577 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Read 67 bytes, read type CACHED, file offset: 0, impl offset: 67/67, impl position: 0, segment: [0, 66]
2026.05.06 08:50:18.717676 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Remaining size to read: 67, read: 67. Resizing buffer to 67
2026.05.06 08:50:18.717910 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(4f35d3dc379904cf70c62240c71a85ac): Key: 4f35d3dc379904cf70c62240c71a85ac. Returning with 67 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 67, current range: [0, 66], file offset of buffer end: 67, file segment state: DOWNLOADED, current write offset: 67, read_type: CACHED, reading until position: 67, started with offset: 0, remaining ranges: [0, 66]
2026.05.06 08:50:18.718878 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/dtv/zjcipqtakuuejtssdwywgiudbclae (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/serialization.json)
2026.05.06 08:50:18.719022 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Reset state for seek to position 0
2026.05.06 08:50:18.719170 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Having 1 file segments to read: [0, 204], current read range: [0, 205)
2026.05.06 08:50:18.719436 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.719707 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 204], key: 0f8745be7660054901be780588901782, state: DOWNLOADED, downloaded size: 205, reserved size: 205, downloader id: None, current write offset: 205, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.719883 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Read 205 bytes, read type CACHED, file offset: 0, impl offset: 205/205, impl position: 0, segment: [0, 204]
2026.05.06 08:50:18.719980 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Remaining size to read: 205, read: 205. Resizing buffer to 205
2026.05.06 08:50:18.720183 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(0f8745be7660054901be780588901782): Key: 0f8745be7660054901be780588901782. Returning with 205 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 205, current range: [0, 204], file offset of buffer end: 205, file segment state: DOWNLOADED, current write offset: 205, read_type: CACHED, reading until position: 205, started with offset: 0, remaining ranges: [0, 204]
2026.05.06 08:50:18.721303 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/vkj/vgfmiarkclwmooungxolbaiivsldy (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/metadata_version.txt)
2026.05.06 08:50:18.721434 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Reset state for seek to position 0
2026.05.06 08:50:18.721598 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:18.721863 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.722069 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: e975169734962c94d801165a31501fba, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.722234 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:18.722332 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:18.722546 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(e975169734962c94d801165a31501fba): Key: e975169734962c94d801165a31501fba. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:18.722804 [ 32555 ] {} <Fatal> BaseDaemon: 11. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:1071: zkutil::ZooKeeper::deleteEphemeralNodeIfContentMatches(String const&, String const&) @ 0x000000003cd4d4fa
2026.05.06 08:50:18.723547 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/jsj/xyihxmffegxddtcontnqlpnwidbrp (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/checksums.txt)
2026.05.06 08:50:18.723709 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Reset state for seek to position 0
2026.05.06 08:50:18.723856 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Having 1 file segments to read: [0, 275], current read range: [0, 276)
2026.05.06 08:50:18.724117 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.724322 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 275], key: c07538018b3df63ed790dfe1d5992011, state: DOWNLOADED, downloaded size: 276, reserved size: 276, downloader id: None, current write offset: 276, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.724492 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Read 276 bytes, read type CACHED, file offset: 0, impl offset: 276/276, impl position: 0, segment: [0, 275]
2026.05.06 08:50:18.724604 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Remaining size to read: 276, read: 276. Resizing buffer to 276
2026.05.06 08:50:18.724802 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(c07538018b3df63ed790dfe1d5992011): Key: c07538018b3df63ed790dfe1d5992011. Returning with 276 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 276, current range: [0, 275], file offset of buffer end: 276, file segment state: DOWNLOADED, current write offset: 276, read_type: CACHED, reading until position: 276, started with offset: 0, remaining ranges: [0, 275]
2026.05.06 08:50:18.725886 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/bxa/xnosofqbvvrmdyictnczhvqeiuwpb (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/n.cmrk2)
2026.05.06 08:50:18.726017 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Reset state for seek to position 0
2026.05.06 08:50:18.726167 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Having 1 file segments to read: [0, 46], current read range: [0, 47)
2026.05.06 08:50:18.726430 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.726659 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 46], key: 33385792238d709413a230095c6af1d8, state: DOWNLOADED, downloaded size: 47, reserved size: 47, downloader id: None, current write offset: 47, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.726820 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Read 47 bytes, read type CACHED, file offset: 0, impl offset: 47/47, impl position: 0, segment: [0, 46]
2026.05.06 08:50:18.726916 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Remaining size to read: 47, read: 47. Resizing buffer to 47
2026.05.06 08:50:18.727129 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(33385792238d709413a230095c6af1d8): Key: 33385792238d709413a230095c6af1d8. Returning with 47 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 47, current range: [0, 46], file offset of buffer end: 47, file segment state: DOWNLOADED, current write offset: 47, read_type: CACHED, reading until position: 47, started with offset: 0, remaining ranges: [0, 46]
2026.05.06 08:50:18.728183 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/xgm/kiwavirgfppuhmpektovwtduervsr (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/count.txt)
2026.05.06 08:50:18.728316 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Reset state for seek to position 0
2026.05.06 08:50:18.728463 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Having 1 file segments to read: [0, 0], current read range: [0, 1)
2026.05.06 08:50:18.728745 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.728964 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 0], key: d96b10f818af9fc56b70c3886e85f9c3, state: DOWNLOADED, downloaded size: 1, reserved size: 1, downloader id: None, current write offset: 1, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.729125 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Read 1 bytes, read type CACHED, file offset: 0, impl offset: 1/1, impl position: 0, segment: [0, 0]
2026.05.06 08:50:18.729217 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Remaining size to read: 1, read: 1. Resizing buffer to 1
2026.05.06 08:50:18.729416 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(d96b10f818af9fc56b70c3886e85f9c3): Key: d96b10f818af9fc56b70c3886e85f9c3. Returning with 1 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 1, current range: [0, 0], file offset of buffer end: 1, file segment state: DOWNLOADED, current write offset: 1, read_type: CACHED, reading until position: 1, started with offset: 0, remaining ranges: [0, 0]
2026.05.06 08:50:18.730180 [ 32281 ] {} <Test> ReadBufferFromRemoteFSGather: Reading from file: s3/ump/ougvhiumadexlpammfrbjimktvxxx (store/2ed/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/tmp-fetch_all_1_1_0_2/default_compression_codec.txt)
2026.05.06 08:50:18.730305 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Reset state for seek to position 0
2026.05.06 08:50:18.730445 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Having 1 file segments to read: [0, 9], current read range: [0, 10)
2026.05.06 08:50:18.730731 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl read range: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.730945 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Current read type: CACHED, read offset: 0, impl offset: 0, impl position: 0, file segment: File segment: [0, 9], key: 1f29f94719cc1b8336e4b3a854c89221, state: DOWNLOADED, downloaded size: 10, reserved size: 10, downloader id: None, current write offset: 10, caller id: None:MergeMutate:32281, kind: Regular, unbound: 0
2026.05.06 08:50:18.731099 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Read 10 bytes, read type CACHED, file offset: 0, impl offset: 10/10, impl position: 0, segment: [0, 9]
2026.05.06 08:50:18.731189 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Remaining size to read: 10, read: 10. Resizing buffer to 10
2026.05.06 08:50:18.731385 [ 32281 ] {} <Test> CachedOnDiskReadBufferFromFile(1f29f94719cc1b8336e4b3a854c89221): Key: 1f29f94719cc1b8336e4b3a854c89221. Returning with 10 bytes, buffer position: 0 (offset: 0, predownloaded: 0), buffer available: 10, current range: [0, 9], file offset of buffer end: 10, file segment state: DOWNLOADED, current write offset: 10, read_type: CACHED, reading until position: 10, started with offset: 0, remaining ranges: [0, 9]
2026.05.06 08:50:18.732024 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845) (Fetcher): Download of part all_1_1_0_2 unique id s3/jsj/xyihxmffegxddtcontnqlpnwidbrp metadata onto disk s3_disk finished.
2026.05.06 08:50:18.732906 [ 32281 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Renaming temporary part tmp-fetch_all_1_1_0_2 to all_1_1_0_2 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
2026.05.06 08:50:18.733340 [ 32281 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): preparePartForCommit: inserting all_1_1_0_2 (state PreActive) into data_parts_indexes
2026.05.06 08:50:18.733473 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Committing part all_1_1_0_2 to zookeeper
2026.05.06 08:50:18.737494 [ 32281 ] {} <Debug> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Undoing transaction (1, 1, 00000000-0000-0000-0000-000000000000). Removing parts: all_1_1_0_2.
2026.05.06 08:50:18.738823 [ 32281 ] {} <Error> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.739455 [ 32281 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2}: Code: 999. Coordination::Exception: Fault injected (before part commit). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<String const&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000003cd0befa
4. ./build_docker/./src/Common/ZooKeeper/IKeeper.cpp:66: Coordination::SimpleFaultInjection::SimpleFaultInjection(double, double, String const&) @ 0x000000003cd09c73
5. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr<DB::IMergeTreeDataPart> const&, std::optional<DB::HardlinkedFiles>, bool) @ 0x00000000388f03b3
6. ./contrib/llvm-project/libcxx/include/vector:961: DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String const&, String const&, bool, unsigned long, std::shared_ptr<zkutil::ZooKeeper>, bool) @ 0x000000003896b1e5
7. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x0000000038942330
8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:170: DB::ReplicatedMergeMutateTaskBase::executeImpl()::$_1::operator()(bool) const @ 0x000000003a162ec3
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000003a15fa63
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:0: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000003a15c14c
11. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x00000000397e749e
12. ./build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:0: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000397e9268
13. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000397f057f
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000202de773
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x00000000202f0da6
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x00000000202f0b5f
17. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
18. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
19. ? @ 0x00007f6e2fbe4ac3
20. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:18.739897 [ 31850 ] {} <Debug> test_13.rmt1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000002 of type MUTATE_PART for part all_0_0_0_2 because recently it has failed. According to exponential backoff policy, put aside this log entry.
2026.05.06 08:50:18.838558 [ 31846 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 4476ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:19.086656 [ 32555 ] {} <Fatal> BaseDaemon: 12. ./build_docker/./src/Databases/DatabaseReplicatedWorker.cpp:0: DB::DatabaseReplicatedDDLWorker::initializeReplication() @ 0x0000000032bed841
2026.05.06 08:50:19.087888 [ 31844 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 69328ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:19.127716 [ 31848 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 1606ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:19.408744 [ 32555 ] {} <Fatal> BaseDaemon: 13. ./build_docker/./src/Databases/DatabaseReplicatedWorker.cpp:0: DB::DatabaseReplicatedDDLWorker::initializeMainThread() @ 0x0000000032beab90
2026.05.06 08:50:33.604771 [ 32192 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_0_0_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:33.605428 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_0_0_0 to all_0_0_0_2
2026.05.06 08:50:33.605502 [ 32178 ] {} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:33.605546 [ 32192 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Zero-copy lock on path /clickhouse/zero_copy/zero_copy_s3/2edd1075-136c-4caa-a6fc-8fb7a8cd3845/all_1_1_0_2/part_exclusive_lock doesn't exist
2026.05.06 08:50:33.605899 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Executing log entry to mutate part all_1_1_0 to all_1_1_0_2
2026.05.06 08:50:33.606214 [ 32285 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_1_1_0_2 (MutateFromLogEntryTask): Mutating part all_1_1_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:33.606535 [ 31817 ] {} <Information> ZooKeeperClient: Finalizing session 5. finalization_started: false, queue_finished: false, reason: 'Operation timeout on Exists /clickhouse/01034_move_partition_from_table_zookeeper_test_15/dst2/log'
2026.05.06 08:50:33.607035 [ 32286 ] {} <Trace> 2edd1075-136c-4caa-a6fc-8fb7a8cd3845::all_0_0_0_2 (MutateFromLogEntryTask): Mutating part all_0_0_0_2 with mutation commands from 1 mutations (0000000000): UPDATE k = 0 WHERE 0
2026.05.06 08:50:33.607046 [ 31817 ] {} <Test> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.607199 [ 31640 ] {} <Information> KeeperTCPHandler: Receiving request for session 5 took 14479 ms
2026.05.06 08:50:33.607212 [ 31817 ] {} <Trace> shard_1.from_1 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.607368 [ 31640 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #5
2026.05.06 08:50:33.607311 [ 31732 ] {} <Warning> test_15.partslost_1 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.607393 [ 31773 ] {} <Warning> test_13.alter_default (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.607451 [ 31745 ] {} <Warning> test_13.table_for_freeze_replicated (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.607349 [ 31726 ] {} <Warning> test_13.table_for_rename_replicated (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.607487 [ 31817 ] {} <Test> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.607700 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:50:33.607731 [ 31745 ] {} <Debug> EphemeralNodeHolder: Cannot remove /test/01417_freeze_partition_verbose_zookeeper_test_13/table_for_freeze_replicated/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.607333 [ 31730 ] {} <Warning> test_15.t_r_ok (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.607403 [ 31757 ] {} <Warning> test_1.insert_dedup_token2 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.607393 [ 31721 ] {} <Warning> test_1.replicated_mutation_table (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.607937 [ 31730 ] {} <Debug> EphemeralNodeHolder: Cannot remove /tables/test_15/t/replicas/r2/is_active since session has been expired
2026.05.06 08:50:33.607996 [ 31745 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Waiting for threads to finish
2026.05.06 08:50:33.607334 [ 31728 ] {} <Warning> test_1.test_without_merge (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.607460 [ 31720 ] {} <Warning> test_1.t (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.607457 [ 31735 ] {} <Warning> test_13.rmt1 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.608110 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:33.608159 [ 31720 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/test/test_1/test_02477/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.608176 [ 31735 ] {} <Debug> EphemeralNodeHolder: Cannot remove /test/02485/test_13/rmt/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.607406 [ 31774 ] {} <Warning> shard_1.from_1 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.608039 [ 31721 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/01414_mutations_and_errors_zookeeper_test_1/mutation_table/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.608231 [ 31720 ] {} <Trace> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Waiting for threads to finish
2026.05.06 08:50:33.608248 [ 31735 ] {} <Trace> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Waiting for threads to finish
2026.05.06 08:50:33.608040 [ 31757 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/test_1/insert_dedup_token/replicas/r2/is_active since session has been expired
2026.05.06 08:50:33.608294 [ 31774 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/from_1_test_uasu1ifx/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.608318 [ 31721 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Waiting for threads to finish
2026.05.06 08:50:33.608333 [ 31732 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/table/test_15_02067_lost/partslost/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.608063 [ 31730 ] {} <Trace> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Waiting for threads to finish
2026.05.06 08:50:33.607818 [ 31726 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/01213_alter_rename_column_zookeeper_long_test_13/table_for_rename_replicated/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.608426 [ 31732 ] {} <Trace> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Waiting for threads to finish
2026.05.06 08:50:33.607816 [ 31817 ] {} <Trace> test_15.t_r_ok (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.608451 [ 31773 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/test_13/test_01079/alter_default/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.608503 [ 31726 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Waiting for threads to finish
2026.05.06 08:50:33.608556 [ 31773 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Waiting for threads to finish
2026.05.06 08:50:33.608557 [ 31817 ] {} <Test> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.608632 [ 32178 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 556.68 GiB.
2026.05.06 08:50:33.608181 [ 31728 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/test_1/test02676_without_merge/replicas/node/is_active since session has been expired
2026.05.06 08:50:33.608675 [ 31817 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.608365 [ 31757 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Waiting for threads to finish
2026.05.06 08:50:33.608733 [ 31817 ] {} <Test> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.608740 [ 31728 ] {} <Trace> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Waiting for threads to finish
2026.05.06 08:50:33.608784 [ 31817 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.608889 [ 31817 ] {} <Test> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.608355 [ 31774 ] {} <Trace> shard_1.from_1 (0c79b048-5607-435e-b6aa-010861f4a826): Waiting for threads to finish
2026.05.06 08:50:33.608889 [ 31808 ] {} <Warning> shard_0.from_0 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.608940 [ 31817 ] {} <Trace> test_13.rmt2 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.609017 [ 31808 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/from_0_test_uasu1ifx/replicas/0/is_active since session has been expired
2026.05.06 08:50:33.609061 [ 31808 ] {} <Trace> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Waiting for threads to finish
2026.05.06 08:50:33.609067 [ 31817 ] {} <Test> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.609117 [ 31817 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.609169 [ 31817 ] {} <Test> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.609218 [ 31817 ] {} <Trace> shard_0.to (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.609280 [ 31817 ] {} <Test> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.609284 [ 31773 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Threads finished
2026.05.06 08:50:33.609330 [ 31817 ] {} <Trace> test_15.t (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.609304 [ 31738 ] {} <Warning> test_13.rmt2 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.609335 [ 31712 ] {} <Warning> shard_0.to (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.609398 [ 31817 ] {} <Test> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.609415 [ 31738 ] {} <Debug> EphemeralNodeHolder: Cannot remove /test/02485/test_13/rmt/replicas/2/is_active since session has been expired
2026.05.06 08:50:33.609406 [ 31691 ] {} <Warning> test_15.t (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.609450 [ 31773 ] {} <Debug> Context: Trying to establish a new connection with ZooKeeper
2026.05.06 08:50:33.609509 [ 31691 ] {} <Debug> EphemeralNodeHolder: Cannot remove /tables/test_15/t/replicas/r1/is_active since session has been expired
2026.05.06 08:50:33.609541 [ 31691 ] {} <Trace> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Waiting for threads to finish
2026.05.06 08:50:33.609525 [ 31726 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Threads finished
2026.05.06 08:50:33.609448 [ 31817 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.609565 [ 31745 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Threads finished
2026.05.06 08:50:33.609486 [ 31712 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/to_test_uasu1ifx/replicas/0/is_active since session has been expired
2026.05.06 08:50:33.609495 [ 31738 ] {} <Trace> test_13.rmt2 (8ef9e61f-45fe-4d95-8c98-3a5c8ab69768): Waiting for threads to finish
2026.05.06 08:50:33.609768 [ 31712 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Waiting for threads to finish
2026.05.06 08:50:33.609858 [ 31757 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Threads finished
2026.05.06 08:50:33.610153 [ 31773 ] {} <Trace> ZooKeeperClient: Adding ZooKeeper host localhost:9181 ([::1]:9181), az: UNKNOWN, priority: 0
2026.05.06 08:50:33.610263 [ 31721 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Threads finished
2026.05.06 08:50:33.610432 [ 32267 ] {} <Test> Poco: Queue size: 0, current threads: 1, threads in pool: 5, current connections: 1
2026.05.06 08:50:33.610498 [ 31640 ] {} <Debug> KeeperTCPHandler: Received close event with xid 2147483647 for session id #5
2026.05.06 08:50:33.610453 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:50:33.609720 [ 31730 ] {} <Trace> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Threads finished
2026.05.06 08:50:33.609674 [ 31817 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.610686 [ 31641 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: [::1]:37832
2026.05.06 08:50:33.610753 [ 31817 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.610837 [ 31817 ] {} <Test> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.610892 [ 31817 ] {} <Trace> shard_1.to (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.610972 [ 31817 ] {} <Test> test_13.rmt1 (2edd1075-136c-4caa-a6fc-8fb7a8cd3845): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.611013 [ 31691 ] {} <Trace> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Threads finished
2026.05.06 08:50:33.611021 [ 31817 ] {} <Trace> test_13.rmt1 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.611073 [ 31817 ] {} <Test> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.611039 [ 31695 ] {} <Warning> shard_1.to (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.611122 [ 31817 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.611197 [ 31817 ] {} <Test> test_1.test_without_merge (920b4c2c-6de3-4d9a-aa58-0f0e78c16a55): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.611230 [ 31690 ] {} <Warning> shard_0.from_1 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.611280 [ 31817 ] {} <Trace> test_1.test_without_merge (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.611195 [ 31695 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/to_test_uasu1ifx/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.611336 [ 31817 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.611350 [ 31690 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/from_1_test_uasu1ifx/replicas/0/is_active since session has been expired
2026.05.06 08:50:33.611386 [ 31817 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.611413 [ 31690 ] {} <Trace> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Waiting for threads to finish
2026.05.06 08:50:33.611384 [ 31695 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Waiting for threads to finish
2026.05.06 08:50:33.611444 [ 31817 ] {} <Test> test_1.t (3788d1f2-ca8f-4730-aff2-1a0f511b50bd): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.611554 [ 31817 ] {} <Trace> test_1.t (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.611635 [ 31817 ] {} <Test> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.611688 [ 31817 ] {} <Trace> test_15.dst (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.611761 [ 31817 ] {} <Test> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.611810 [ 31817 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.611817 [ 31776 ] {} <Warning> test_15.dst (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.611895 [ 31688 ] {} <Warning> test_1.insert_dedup_token1 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.611882 [ 31817 ] {} <Test> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.611953 [ 31688 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/test_1/insert_dedup_token/replicas/r1/is_active since session has been expired
2026.05.06 08:50:33.611982 [ 31688 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Waiting for threads to finish
2026.05.06 08:50:33.611976 [ 31817 ] {} <Trace> test_15.src (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.611996 [ 31776 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/01034_move_partition_from_table_zookeeper_test_15/dst2/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.612080 [ 31776 ] {} <Trace> test_15.dst (91844baf-1583-4a3f-9f40-e6d36837eb7d): Waiting for threads to finish
2026.05.06 08:50:33.612103 [ 31686 ] {} <Warning> test_15.src (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.612090 [ 31817 ] {} <Test> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.612182 [ 31817 ] {} <Trace> shard_1.from_0 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.612161 [ 31686 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/01034_move_partition_from_table_zookeeper_test_15/src1/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.612254 [ 31686 ] {} <Trace> test_15.src (c8e903e3-5254-4279-92c8-4041d14dee92): Waiting for threads to finish
2026.05.06 08:50:33.612275 [ 31817 ] {} <Test> test_15.partslost_1 (19e8a1ba-691b-4ad1-b0cb-44600f05ccee): Received event for expired session. Waking up restarting thread
2026.05.06 08:50:33.612326 [ 31817 ] {} <Trace> test_15.partslost_1 (ReplicatedMergeTreeRestartingThread): Starting restating thread, schedule: true
2026.05.06 08:50:33.612328 [ 31714 ] {} <Warning> shard_1.from_0 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2026.05.06 08:50:33.612499 [ 31714 ] {} <Debug> EphemeralNodeHolder: Cannot remove /clickhouse/tables/from_0_test_uasu1ifx/replicas/1/is_active since session has been expired
2026.05.06 08:50:33.612579 [ 31714 ] {} <Trace> shard_1.from_0 (61a4e379-0c82-4c81-820a-4097214aef2e): Waiting for threads to finish
2026.05.06 08:50:33.612893 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:33.617606 [ 31641 ] {} <Information> KeeperTCPHandler: Requesting session ID for the new client
2026.05.06 08:50:33.617696 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query to stage Complete
2026.05.06 08:50:33.617825 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 0
2026.05.06 08:50:33.618054 [ 31640 ] {} <Debug> KeeperTCPHandler: Session #5 successfully closed
2026.05.06 08:50:33.618126 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:33.618441 [ 32262 ] {} <Debug> KeeperStateMachine: Session ID response 6 with timeout 30000
2026.05.06 08:50:33.618765 [ 31641 ] {} <Information> KeeperTCPHandler: Received session ID 6
2026.05.06 08:50:33.618814 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Planner: Query from stage FetchColumns to stage Complete
2026.05.06 08:50:33.619665 [ 31773 ] {} <Information> ZooKeeperClient: Connected to ZooKeeper at [::1]:9181 with session_id 6
2026.05.06 08:50:33.621022 [ 31773 ] {} <Information> ZooKeeperClient: Keeper feature flag FILTERED_LIST: disabled
2026.05.06 08:50:33.621182 [ 31773 ] {} <Information> ZooKeeperClient: Keeper feature flag MULTI_READ: disabled
2026.05.06 08:50:33.621245 [ 31773 ] {} <Information> ZooKeeperClient: Keeper feature flag CHECK_NOT_EXISTS: enabled
2026.05.06 08:50:33.621308 [ 31773 ] {} <Information> ZooKeeperClient: Keeper feature flag CREATE_IF_NOT_EXISTS: enabled
2026.05.06 08:50:33.621398 [ 31773 ] {} <Trace> ZooKeeper: Initialized, hosts: localhost:9181
2026.05.06 08:50:33.621859 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 75
2026.05.06 08:50:33.622282 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:33.623041 [ 31773 ] {} <Information> ZooKeeperClient: ZK send fault: 0.002%
2026.05.06 08:50:33.623201 [ 31773 ] {} <Information> ZooKeeperClient: ZK recv fault: 0.002%
2026.05.06 08:50:33.623380 [ 31773 ] {} <Information> ZooKeeperClient: ZK send sleep: 0.001% -> 10000ms
2026.05.06 08:50:33.623553 [ 31773 ] {} <Debug> Context: Establishing a new connection with ZooKeeper took 14 ms
2026.05.06 08:50:33.623750 [ 31773 ] {} <Debug> test_13.alter_default (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.623815 [ 31726 ] {} <Debug> test_13.table_for_rename_replicated (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.623910 [ 31757 ] {} <Debug> test_1.insert_dedup_token2 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.623875 [ 31745 ] {} <Debug> test_13.table_for_freeze_replicated (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.623975 [ 31721 ] {} <Debug> test_1.replicated_mutation_table (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.624013 [ 31730 ] {} <Debug> test_15.t_r_ok (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.624037 [ 31691 ] {} <Debug> test_15.t (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.625657 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 355
2026.05.06 08:50:33.626350 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:33.626437 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 6, bytes: 2213
2026.05.06 08:50:33.629200 [ 32244 ] {} <Test> KeeperLogStore: Flushing 6 logs
2026.05.06 08:50:33.631984 [ 31757 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.632082 [ 31730 ] {} <Trace> test_15.t_r_ok (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.632057 [ 31745 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.632158 [ 31726 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.632171 [ 31721 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.632194 [ 31691 ] {} <Trace> test_15.t (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.632444 [ 31773 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.632980 [ 31726 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Adding part 20191001_0_0_0_1 to current and virtual parts
2026.05.06 08:50:33.632992 [ 31730 ] {} <Trace> test_15.t_r_ok (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.633087 [ 31757 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Adding part all_1_1_0 to current and virtual parts
2026.05.06 08:50:33.633132 [ 31721 ] {} <Test> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Adding part 20191002_1_1_0_4 to current and virtual parts
2026.05.06 08:50:33.633198 [ 31757 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Adding part all_2_2_0 to current and virtual parts
2026.05.06 08:50:33.633209 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 5_2_2_0 to current and virtual parts
2026.05.06 08:50:33.633199 [ 31726 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Adding part 20191003_0_0_0_1 to current and virtual parts
2026.05.06 08:50:33.633250 [ 31730 ] {} <Debug> test_15.t_r_ok (ReplicatedMergeTreeQueue): Loading queue from /tables/test_15/t/replicas/r2/queue
2026.05.06 08:50:33.633281 [ 31691 ] {} <Trace> test_15.t (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.633308 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 3_3_3_0 to current and virtual parts
2026.05.06 08:50:33.633226 [ 31721 ] {} <Test> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Adding part 20191002_0_0_0_4 to current and virtual parts
2026.05.06 08:50:33.633281 [ 31757 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Adding part all_0_0_0 to current and virtual parts
2026.05.06 08:50:33.633333 [ 31726 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Adding part 20191002_0_0_0_1 to current and virtual parts
2026.05.06 08:50:33.633384 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 2_0_0_0 to current and virtual parts
2026.05.06 08:50:33.633386 [ 31691 ] {} <Debug> test_15.t (ReplicatedMergeTreeQueue): Loading queue from /tables/test_15/t/replicas/r1/queue
2026.05.06 08:50:33.633343 [ 31773 ] {} <Test> test_13.alter_default (ReplicatedMergeTreeQueue): Adding part all_0_0_0_3 to current and virtual parts
2026.05.06 08:50:33.633460 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 3_2_2_0 to current and virtual parts
2026.05.06 08:50:33.633425 [ 31757 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Adding part all_0_2_1 to current and virtual parts
2026.05.06 08:50:33.633427 [ 31726 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.633551 [ 31773 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.633398 [ 31721 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.633595 [ 31757 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.633545 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 1_0_0_0 to current and virtual parts
2026.05.06 08:50:33.633644 [ 31726 ] {} <Debug> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/01213_alter_rename_column_zookeeper_long_test_13/table_for_rename_replicated/replicas/1/queue
2026.05.06 08:50:33.633670 [ 31757 ] {} <Debug> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/test_1/insert_dedup_token/replicas/r2/queue
2026.05.06 08:50:33.633676 [ 31721 ] {} <Debug> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/01414_mutations_and_errors_zookeeper_test_1/mutation_table/replicas/1/queue
2026.05.06 08:50:33.633667 [ 31773 ] {} <Debug> test_13.alter_default (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/test_13/test_01079/alter_default/replicas/1/queue
2026.05.06 08:50:33.633685 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 8_0_0_0 to current and virtual parts
2026.05.06 08:50:33.633790 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 4_0_0_0 to current and virtual parts
2026.05.06 08:50:33.633858 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 6_0_0_0 to current and virtual parts
2026.05.06 08:50:33.633931 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 7_0_0_0 to current and virtual parts
2026.05.06 08:50:33.634009 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 9_0_0_0 to current and virtual parts
2026.05.06 08:50:33.634090 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Adding part 0_0_0_0 to current and virtual parts
2026.05.06 08:50:33.634093 [ 31730 ] {} <Debug> test_15.t_r_ok (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.634103 [ 31691 ] {} <Debug> test_15.t (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.634157 [ 31726 ] {} <Debug> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.634212 [ 31757 ] {} <Debug> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.634161 [ 31745 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.634304 [ 31721 ] {} <Debug> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.634353 [ 31773 ] {} <Debug> test_13.alter_default (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.634334 [ 31745 ] {} <Debug> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Loading queue from /test/01417_freeze_partition_verbose_zookeeper_test_13/table_for_freeze_replicated/replicas/1/queue
2026.05.06 08:50:33.634812 [ 31726 ] {} <Debug> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Updating strategy picker state
2026.05.06 08:50:33.634789 [ 31730 ] {} <Debug> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Updating strategy picker state
2026.05.06 08:50:33.634904 [ 31757 ] {} <Debug> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Updating strategy picker state
2026.05.06 08:50:33.634978 [ 31691 ] {} <Debug> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Updating strategy picker state
2026.05.06 08:50:33.635045 [ 31721 ] {} <Debug> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Updating strategy picker state
2026.05.06 08:50:33.635117 [ 31745 ] {} <Debug> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.635182 [ 31773 ] {} <Debug> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Updating strategy picker state
2026.05.06 08:50:33.635844 [ 31745 ] {} <Debug> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Updating strategy picker state
2026.05.06 08:50:33.636180 [ 31726 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.636340 [ 31721 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.636516 [ 31773 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.636606 [ 31757 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.636738 [ 31730 ] {} <Debug> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Strategy picker state updated, current replica: 1, active replicas: [r1, r2]
2026.05.06 08:50:33.636786 [ 31691 ] {} <Debug> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Strategy picker state updated, current replica: 0, active replicas: [r1, r2]
2026.05.06 08:50:33.636895 [ 31691 ] {} <Trace> test_15.t (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.636897 [ 31730 ] {} <Trace> test_15.t_r_ok (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.637026 [ 31745 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.639589 [ 31726 ] {} <Information> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Loading 1 mutation entries: 0000000000 - 0000000000
2026.05.06 08:50:33.639589 [ 31721 ] {} <Information> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Loading 1 mutation entries: 0000000002 - 0000000002
2026.05.06 08:50:33.639774 [ 31773 ] {} <Information> test_13.alter_default (ReplicatedMergeTreeQueue): Loading 3 mutation entries: 0000000000 - 0000000002
2026.05.06 08:50:33.641521 [ 31726 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Adding mutation 0000000000 for 3 partitions (data versions: 20191001 = 1; 20191002 = 1; 20191003 = 1; )
2026.05.06 08:50:33.641554 [ 32475 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregating
2026.05.06 08:50:33.641599 [ 31721 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Adding mutation 0000000002 for 1 partitions (data versions: 20191002 = 4; )
2026.05.06 08:50:33.641734 [ 32475 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Aggregation method: without_key
2026.05.06 08:50:33.642047 [ 32475 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> AggregatingTransform: Aggregated. 1 to 1 rows (from 24.00 B) in 0.006993549 sec. (142.989 rows/sec., 3.35 KiB/sec.)
2026.05.06 08:50:33.642138 [ 32475 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> Aggregator: Merging aggregated data
2026.05.06 08:50:33.642572 [ 31773 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Adding mutation 0000000000 for 1 partitions (data versions: all = 1; )
2026.05.06 08:50:33.642715 [ 31773 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Adding mutation 0000000001 for 1 partitions (data versions: all = 2; )
2026.05.06 08:50:33.642820 [ 31773 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Adding mutation 0000000002 for 1 partitions (data versions: all = 3; )
2026.05.06 08:50:33.644611 [ 31721 ] {} <Test> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Removing part 20191002_0_0_0_4 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.644717 [ 31757 ] {} <Test> test_1.insert_dedup_token2 (ReplicatedMergeTreeQueue): Removing part all_0_2_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.644747 [ 31721 ] {} <Test> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Removing part 20191002_1_1_0_4 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.644845 [ 31726 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Removing part 20191001_0_0_0_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.644959 [ 31726 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Removing part 20191002_0_0_0_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.645037 [ 31726 ] {} <Test> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Removing part 20191003_0_0_0_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.645533 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 0_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.645629 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 1_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.645706 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 2_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.645773 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 3_2_2_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.645838 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 3_3_3_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.645904 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 4_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.645970 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 5_2_2_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.646036 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 6_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.646102 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 7_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.646152 [ 31795 ] {} <Trace> test_15.t_r_ok (3d94e4c2-328b-41c2-af2d-74c0c4598708): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.646168 [ 31743 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeQueue): Marking mutation 0000000002 done because it is <= mutation_pointer (0000000002)
2026.05.06 08:50:33.646168 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 8_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.646272 [ 31745 ] {} <Test> test_13.table_for_freeze_replicated (ReplicatedMergeTreeQueue): Removing part 9_0_0_0 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.646299 [ 31721 ] {} <Debug> test_1.replicated_mutation_table (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.646439 [ 31757 ] {} <Debug> test_1.insert_dedup_token2 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.646516 [ 31792 ] {} <Test> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3) (MergerMutator): Nothing to merge in partition 20191002: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.646612 [ 32189 ] {} <Test> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.646685 [ 31730 ] {} <Debug> test_15.t_r_ok (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.646699 [ 31792 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:50:33.646751 [ 32189 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:50:33.646893 [ 32189 ] {} <Trace> test_1.insert_dedup_token2 (0486a7ae-5268-42e9-bce5-dd91cd54e355): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.647053 [ 31795 ] {} <Trace> test_15.t (b7ee17fe-d872-4ddc-8c1c-ede485e51abe): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.647201 [ 31824 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeQueue): Marking mutation 0000000000 done because it is <= mutation_pointer (0000000000)
2026.05.06 08:50:33.647242 [ 31691 ] {} <Debug> test_15.t (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.647382 [ 31773 ] {} <Test> test_13.alter_default (ReplicatedMergeTreeQueue): Removing part all_0_0_0_3 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.647520 [ 32171 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Nothing to merge in partition 20191001: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.647650 [ 32171 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Nothing to merge in partition 20191002: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.647725 [ 32171 ] {} <Test> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Nothing to merge in partition 20191003: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.647809 [ 31726 ] {} <Debug> test_13.table_for_rename_replicated (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.647824 [ 32171 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67) (MergerMutator): Checked 3 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:50:33.648237 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Trace> MutateTask: Mutating part 1_1_1_0 to mutation version 3
2026.05.06 08:50:33.649113 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MutationsInterpreter(test_1.t1): Will use old analyzer to prepare mutation
2026.05.06 08:50:33.649554 [ 31745 ] {} <Debug> test_13.table_for_freeze_replicated (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.649890 [ 32165 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 0: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.650007 [ 32165 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 1: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.650081 [ 32165 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 2: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.650166 [ 32165 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 3: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.650238 [ 32165 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 4: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.650310 [ 32165 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 5: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.650381 [ 32165 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 6: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.650454 [ 32165 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 7: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.650543 [ 32165 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 8: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.650600 [ 31773 ] {} <Debug> test_13.alter_default (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.650620 [ 32157 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Marking mutation 0000000000 done because it is <= mutation_pointer (0000000002)
2026.05.06 08:50:33.650615 [ 32165 ] {} <Test> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Nothing to merge in partition 9: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.650727 [ 32157 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Marking mutation 0000000001 done because it is <= mutation_pointer (0000000002)
2026.05.06 08:50:33.650794 [ 32157 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeQueue): Marking mutation 0000000002 done because it is <= mutation_pointer (0000000002)
2026.05.06 08:50:33.650796 [ 32165 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a) (MergerMutator): Checked 10 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:50:33.650985 [ 32165 ] {} <Trace> test_13.table_for_freeze_replicated (2da687b0-0ed2-46cf-b873-c6ccbcbd0f0a): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.651047 [ 32155 ] {} <Test> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.651182 [ 32155 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:50:33.651402 [ 31713 ] {} <Trace> test_15.t_r_ok (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 19289ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:33.651649 [ 32190 ] {} <Trace> test_1.insert_dedup_token2 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 19288ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:33.651647 [ 32186 ] {} <Trace> test_15.t (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 19289ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:33.652908 [ 32169 ] {} <Trace> test_13.table_for_rename_replicated (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 17851ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:33.656799 [ 31792 ] {} <Trace> test_1.replicated_mutation_table (35bad05d-d8fe-4647-a804-b081c3be65d3): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.656810 [ 31727 ] {} <Trace> test_1.replicated_mutation_table (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 18376ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:33.656923 [ 32171 ] {} <Trace> test_13.table_for_rename_replicated (608153d5-71f1-4052-9b53-8112203d1f67): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.658192 [ 32151 ] {} <Trace> test_13.alter_default (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 14570ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:33.658334 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 1_1_1_0, total 4 rows starting from the beginning of the part
2026.05.06 08:50:33.658342 [ 32166 ] {} <Trace> test_13.table_for_freeze_replicated (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 15531ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:33.658456 [ 32155 ] {} <Trace> test_13.alter_default (9d1f8fdb-3bcc-4e27-9f23-fbdf765502c1): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.659096 [ 32384 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Test> MergeTreeMarksLoader: Loading marks from path data.cmrk3
2026.05.06 08:50:33.672266 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> MutatePlainMergeTreeTask: Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW) (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:33.675814 [ 32282 ] {bed21b2e-93a1-4da1-ba17-fc452427a4b6::1_1_1_0_3} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 160. DB::Exception: The maximum sleep time is 3000000 microseconds. Requested: 12000000 microseconds per block (of size 4): while executing 'FUNCTION if(and(equals(p, 1), equals(_partition_id, '1')) :: 5, _CAST(sleepEachRow(3), 'UInt64') :: 3, i :: 1) -> if(and(equals(p, 1), equals(_partition_id, '1')), _CAST(sleepEachRow(3), 'UInt64'), i) UInt64 : 4'. (TOO_SLOW), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. DB::Exception::Exception<unsigned long const&, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long const&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, unsigned long const&, unsigned long&, unsigned long&) @ 0x000000001c0c4d17
4. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000001c0c495d
5. DB::FunctionSleep<(DB::FunctionSleepVariant)1>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c0c30d1
6. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
7. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
8. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
9. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
10. DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000083d603f
11. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ColumnFunction::reduce() const @ 0x0000000037583450
12. ./build_docker/./src/Columns/ColumnFunction.cpp:0: DB::ColumnFunction::reduce() const @ 0x00000000375826e4
13. ./build_docker/./src/Columns/MaskOperations.cpp:0: DB::maskedExecute(DB::ColumnWithTypeAndName&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::MaskInfo const&) @ 0x000000003819aa45
14. DB::(anonymous namespace)::FunctionIf::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000125c3da6
15. DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000083d77db
16. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60286a
17. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60463f
18. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000b60a41e
19. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:124: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x0000000033b51e9e
20. ./build_docker/./src/Processors/Transforms/ExpressionTransform.cpp:0: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000003b966a24
21. ./contrib/llvm-project/libcxx/include/__utility/swap.h:35: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000020ad4ea4
22. ./build_docker/./src/Processors/ISimpleTransform.cpp:99: DB::ISimpleTransform::work() @ 0x000000003afa146b
23. ./contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x000000003b0117cd
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003afe0706
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:150: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000003afdf3e4
26. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000003b021672
27. ./build_docker/./src/Processors/Executors/PullingPipelineExecutor.cpp:0: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000003b021f44
28. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1322: DB::PartMergerWriter::mutateOriginalPartAndPrepareProjections() @ 0x0000000039fb557b
29. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::PartMergerWriter::execute() @ 0x000000003a0308a2
30. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:1463: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000003a020c52
31. ./build_docker/./src/Storages/MergeTree/MutateTask.cpp:0: DB::MutateTask::execute() @ 0x0000000039fc1bdf
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:33.676312 [ 32282 ] {} <Debug> MemoryTracker: Peak memory usage background process to apply mutate/merge in table: 2.08 MiB.
2026.05.06 08:50:33.790809 [ 32382 ] {} <Error> ZooKeeperClient: Code: 999. Coordination::Exception: Operation timeout (deadline of 10000 ms already expired) for path: /clickhouse/01034_move_partition_from_table_zookeeper_test_15/dst2/log. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008381b9d
3. ./src/Common/Exception.h:0: Coordination::Exception::Exception<int&, String>(Coordination::Error, FormatStringHelperImpl<std::type_identity<int&>::type, std::type_identity<String>::type>, int&, String&&) @ 0x000000003ce6ca61
4. ./build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:789: Coordination::ZooKeeper::receiveThread() @ 0x000000003ce4fa19
5. ./build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<zkutil::ShuffleHost, std::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector<zkutil::ShuffleHost, std::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003ce67faf
6. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
7. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
8. ? @ 0x00007f6e2fbe4ac3
9. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:33.792398 [ 32445 ] {} <Debug> DDLWorker(test_11): Scheduling tasks
2026.05.06 08:50:33.792447 [ 32440 ] {} <Debug> DDLWorker(test_5): Scheduling tasks
2026.05.06 08:50:33.876583 [ 32177 ] {} <Error> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): void DB::ReplicatedMergeTreeCleanupThread::run(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. ./src/Common/Exception.h:0: _ZN12Coordination9ExceptionC2IRA16_KcQsr3stdE16is_convertible_vIT_NSt3__112basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEEEEEOS5_NS_5ErrorE @ 0x000000003cdcc9f1
3. ./src/Common/ZooKeeper/IKeeper.h:509: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x000000003ce555de
4. ./build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:1365: Coordination::ZooKeeper::exists(String const&, std::function<void (Coordination::ExistsResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x000000003ce5f332
5. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:0: zkutil::ZooKeeper::asyncTryExistsNoThrow(String const&, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000003cd32121
6. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000003cd31241
7. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:598: zkutil::ZooKeeper::existsWatch(String const&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000003cd32610
8. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:582: zkutil::ZooKeeper::exists(String const&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&) @ 0x000000003cd29411
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:203: DB::ReplicatedMergeTreeCleanupThread::clearOldLogs() @ 0x000000003a17ec69
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:0: DB::ReplicatedMergeTreeCleanupThread::iterate() @ 0x000000003a17c5dc
11. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:50: DB::ReplicatedMergeTreeCleanupThread::run() @ 0x000000003a179038
12. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000003a1a1c1f
13. ./src/Common/Stopwatch.h:72: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000031acbc92
14. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000031ad6fb3
15. ./build_docker/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000031ad8c4f
16. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
17. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
18. ? @ 0x00007f6e2fbe4ac3
19. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:33.876781 [ 32177 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Execution took 272 ms.
2026.05.06 08:50:33.876811 [ 32110 ] {} <Error> shard_0.to (ReplicatedMergeTreeCleanupThread): void DB::ReplicatedMergeTreeCleanupThread::run(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. ./src/Common/Exception.h:0: _ZN12Coordination9ExceptionC2IRA16_KcQsr3stdE16is_convertible_vIT_NSt3__112basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEEEEEOS5_NS_5ErrorE @ 0x000000003cdcc9f1
3. ./src/Common/ZooKeeper/IKeeper.h:509: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x000000003ce555de
4. ./build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:1365: Coordination::ZooKeeper::exists(String const&, std::function<void (Coordination::ExistsResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x000000003ce5f332
5. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:0: zkutil::ZooKeeper::asyncTryExistsNoThrow(String const&, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000003cd32121
6. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000003cd31241
7. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:598: zkutil::ZooKeeper::existsWatch(String const&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000003cd32610
8. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:582: zkutil::ZooKeeper::exists(String const&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&) @ 0x000000003cd29411
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:203: DB::ReplicatedMergeTreeCleanupThread::clearOldLogs() @ 0x000000003a17ec69
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:0: DB::ReplicatedMergeTreeCleanupThread::iterate() @ 0x000000003a17c5dc
11. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:50: DB::ReplicatedMergeTreeCleanupThread::run() @ 0x000000003a179038
12. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000003a1a1c1f
13. ./src/Common/Stopwatch.h:72: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000031acbc92
14. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000031ad6fb3
15. ./build_docker/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000031ad8c4f
16. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
17. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
18. ? @ 0x00007f6e2fbe4ac3
19. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:33.877052 [ 31740 ] {} <Error> shard_1.to (ReplicatedMergeTreeCleanupThread): void DB::ReplicatedMergeTreeCleanupThread::run(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. ./src/Common/Exception.h:0: _ZN12Coordination9ExceptionC2IRA16_KcQsr3stdE16is_convertible_vIT_NSt3__112basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEEEEEOS5_NS_5ErrorE @ 0x000000003cdcc9f1
3. ./src/Common/ZooKeeper/IKeeper.h:509: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x000000003ce555de
4. ./build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:1365: Coordination::ZooKeeper::exists(String const&, std::function<void (Coordination::ExistsResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x000000003ce5f332
5. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:0: zkutil::ZooKeeper::asyncTryExistsNoThrow(String const&, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000003cd32121
6. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000003cd31241
7. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:598: zkutil::ZooKeeper::existsWatch(String const&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000003cd32610
8. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:582: zkutil::ZooKeeper::exists(String const&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&) @ 0x000000003cd29411
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:203: DB::ReplicatedMergeTreeCleanupThread::clearOldLogs() @ 0x000000003a17ec69
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:0: DB::ReplicatedMergeTreeCleanupThread::iterate() @ 0x000000003a17c5dc
11. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:50: DB::ReplicatedMergeTreeCleanupThread::run() @ 0x000000003a179038
12. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000003a1a1c1f
13. ./src/Common/Stopwatch.h:72: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000031acbc92
14. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000031ad6fb3
15. ./build_docker/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000031ad8c4f
16. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
17. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
18. ? @ 0x00007f6e2fbe4ac3
19. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:33.877207 [ 31690 ] {} <Trace> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Threads finished
2026.05.06 08:50:33.877132 [ 32093 ] {} <Error> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): void DB::ReplicatedMergeTreeCleanupThread::run(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. ./src/Common/Exception.h:0: _ZN12Coordination9ExceptionC2IRA16_KcQsr3stdE16is_convertible_vIT_NSt3__112basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEEEEEOS5_NS_5ErrorE @ 0x000000003cdcc9f1
3. ./src/Common/ZooKeeper/IKeeper.h:509: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x000000003ce555de
4. ./build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:1365: Coordination::ZooKeeper::exists(String const&, std::function<void (Coordination::ExistsResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x000000003ce5f332
5. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:0: zkutil::ZooKeeper::asyncTryExistsNoThrow(String const&, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000003cd32121
6. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000003cd31241
7. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:598: zkutil::ZooKeeper::existsWatch(String const&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000003cd32610
8. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:582: zkutil::ZooKeeper::exists(String const&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&) @ 0x000000003cd29411
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:203: DB::ReplicatedMergeTreeCleanupThread::clearOldLogs() @ 0x000000003a17ec69
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:0: DB::ReplicatedMergeTreeCleanupThread::iterate() @ 0x000000003a17c5dc
11. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:50: DB::ReplicatedMergeTreeCleanupThread::run() @ 0x000000003a179038
12. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000003a1a1c1f
13. ./src/Common/Stopwatch.h:72: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000031acbc92
14. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000031ad6fb3
15. ./build_docker/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000031ad8c4f
16. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
17. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
18. ? @ 0x00007f6e2fbe4ac3
19. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:33.877219 [ 32110 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Execution took 271 ms.
2026.05.06 08:50:33.877223 [ 31724 ] {} <Error> test_1.insert_dedup_token1 (ReplicatedMergeTreeCleanupThread): void DB::ReplicatedMergeTreeCleanupThread::run(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000405306e9
1. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000200393f3
2. ./src/Common/Exception.h:0: _ZN12Coordination9ExceptionC2IRA16_KcQsr3stdE16is_convertible_vIT_NSt3__112basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEEEEEOS5_NS_5ErrorE @ 0x000000003cdcc9f1
3. ./src/Common/ZooKeeper/IKeeper.h:509: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x000000003ce555de
4. ./build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:1365: Coordination::ZooKeeper::exists(String const&, std::function<void (Coordination::ExistsResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x000000003ce5f332
5. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:0: zkutil::ZooKeeper::asyncTryExistsNoThrow(String const&, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000003cd32121
6. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000003cd31241
7. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:598: zkutil::ZooKeeper::existsWatch(String const&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000003cd32610
8. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:582: zkutil::ZooKeeper::exists(String const&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&) @ 0x000000003cd29411
9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:203: DB::ReplicatedMergeTreeCleanupThread::clearOldLogs() @ 0x000000003a17ec69
10. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:0: DB::ReplicatedMergeTreeCleanupThread::iterate() @ 0x000000003a17c5dc
11. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:50: DB::ReplicatedMergeTreeCleanupThread::run() @ 0x000000003a179038
12. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000003a1a1c1f
13. ./src/Common/Stopwatch.h:72: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000031acbc92
14. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000031ad6fb3
15. ./build_docker/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000031ad8c4f
16. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x00000000202d75dd
17. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000202e9b24
18. ? @ 0x00007f6e2fbe4ac3
19. ? @ 0x00007f6e2fc76850
 (version 24.8.14.10546.altinitytest (altinity build))
2026.05.06 08:50:33.877312 [ 31690 ] {} <Debug> shard_0.from_1 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.877396 [ 32093 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Execution took 271 ms.
2026.05.06 08:50:33.877302 [ 31740 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Execution took 269 ms.
2026.05.06 08:50:33.877454 [ 31724 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeCleanupThread): Execution took 269 ms.
2026.05.06 08:50:33.877840 [ 31712 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Threads finished
2026.05.06 08:50:33.877917 [ 31808 ] {} <Trace> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Threads finished
2026.05.06 08:50:33.877938 [ 31712 ] {} <Debug> shard_0.to (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.877989 [ 31808 ] {} <Debug> shard_0.from_0 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.878047 [ 31688 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Threads finished
2026.05.06 08:50:33.878133 [ 31688 ] {} <Debug> test_1.insert_dedup_token1 (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.878164 [ 31695 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Threads finished
2026.05.06 08:50:33.878247 [ 31695 ] {} <Debug> shard_1.to (ReplicatedMergeTreeRestartingThread): Trying to start replica up
2026.05.06 08:50:33.879150 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 1, bytes: 324
2026.05.06 08:50:33.879935 [ 32244 ] {} <Test> KeeperLogStore: Flushing 1 logs
2026.05.06 08:50:33.880032 [ 32239 ] {} <Trace> KeeperDispatcher: Processing requests batch, size: 4, bytes: 1296
2026.05.06 08:50:33.881880 [ 32244 ] {} <Test> KeeperLogStore: Flushing 4 logs
2026.05.06 08:50:33.883171 [ 31690 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.883230 [ 31712 ] {} <Trace> shard_0.to (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.883326 [ 31808 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.883391 [ 31688 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.883538 [ 31695 ] {} <Trace> shard_1.to (ReplicatedMergeTreeQueue): Initializing parts in queue
2026.05.06 08:50:33.883782 [ 31690 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.883865 [ 31690 ] {} <Debug> shard_0.from_1 (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/from_1_test_uasu1ifx/replicas/0/queue
2026.05.06 08:50:33.883875 [ 31712 ] {} <Test> shard_0.to (ReplicatedMergeTreeQueue): Adding part all_0_3_1 to current and virtual parts
2026.05.06 08:50:33.883915 [ 31808 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.883952 [ 31712 ] {} <Trace> shard_0.to (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.883980 [ 31808 ] {} <Debug> shard_0.from_0 (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/from_0_test_uasu1ifx/replicas/0/queue
2026.05.06 08:50:33.884016 [ 31688 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Adding part all_2_2_0 to current and virtual parts
2026.05.06 08:50:33.884024 [ 31712 ] {} <Debug> shard_0.to (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/to_test_uasu1ifx/replicas/0/queue
2026.05.06 08:50:33.884111 [ 31688 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Adding part all_1_1_0 to current and virtual parts
2026.05.06 08:50:33.884118 [ 31695 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Adding part all_0_3_1 to current and virtual parts
2026.05.06 08:50:33.884178 [ 31695 ] {} <Trace> shard_1.to (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.884181 [ 31688 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Adding part all_0_2_1 to current and virtual parts
2026.05.06 08:50:33.884226 [ 31695 ] {} <Debug> shard_1.to (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/to_test_uasu1ifx/replicas/1/queue
2026.05.06 08:50:33.884252 [ 31688 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Adding part all_0_0_0 to current and virtual parts
2026.05.06 08:50:33.884307 [ 31688 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Queue initialized
2026.05.06 08:50:33.884385 [ 31688 ] {} <Debug> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Loading queue from /clickhouse/tables/test_1/insert_dedup_token/replicas/r1/queue
2026.05.06 08:50:33.884403 [ 31690 ] {} <Debug> shard_0.from_1 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.884444 [ 31808 ] {} <Debug> shard_0.from_0 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.884647 [ 31712 ] {} <Debug> shard_0.to (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.884728 [ 31695 ] {} <Debug> shard_1.to (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.884934 [ 31688 ] {} <Debug> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Having 0 queue entries to load, 0 entries already loaded.
2026.05.06 08:50:33.884986 [ 31690 ] {} <Debug> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Updating strategy picker state
2026.05.06 08:50:33.885032 [ 31808 ] {} <Debug> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Updating strategy picker state
2026.05.06 08:50:33.885279 [ 31712 ] {} <Debug> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Updating strategy picker state
2026.05.06 08:50:33.885336 [ 31695 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Updating strategy picker state
2026.05.06 08:50:33.885424 [ 31688 ] {} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Updating strategy picker state
2026.05.06 08:50:33.886581 [ 31690 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.886631 [ 31808 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.886749 [ 31712 ] {} <Debug> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Strategy picker state updated, current replica: 0, active replicas: [0, 1]
2026.05.06 08:50:33.886788 [ 31695 ] {} <Debug> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Strategy picker state updated, current replica: 1, active replicas: [0, 1]
2026.05.06 08:50:33.886829 [ 31695 ] {} <Trace> shard_1.to (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.886831 [ 31712 ] {} <Trace> shard_0.to (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.886996 [ 31688 ] {} <Debug> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Strategy picker state updated, current replica: 0, active replicas: [r1, r2]
2026.05.06 08:50:33.887069 [ 31688 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Loaded queue
2026.05.06 08:50:33.890962 [ 31695 ] {} <Test> shard_1.to (ReplicatedMergeTreeQueue): Removing part all_0_3_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.891748 [ 31688 ] {} <Test> test_1.insert_dedup_token1 (ReplicatedMergeTreeQueue): Removing part all_0_2_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.891805 [ 31712 ] {} <Test> shard_0.to (ReplicatedMergeTreeQueue): Removing part all_0_3_1 from mutations (remove_part: false, remove_covered_parts: true)
2026.05.06 08:50:33.891916 [ 31695 ] {} <Debug> shard_1.to (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.891970 [ 31695 ] {} <Trace> shard_1.to (ReplicatedMergeTreeRestartingThread): Execution took 280 ms.
2026.05.06 08:50:33.892104 [ 32130 ] {} <Test> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.892261 [ 32130 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:50:33.892408 [ 32130 ] {} <Trace> shard_1.to (107d7a0f-1932-4bf8-8e25-77087043845f): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.892744 [ 31808 ] {} <Debug> shard_0.from_0 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.892836 [ 31808 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeRestartingThread): Execution took 283 ms.
2026.05.06 08:50:33.892855 [ 32106 ] {} <Trace> shard_0.from_0 (82bb6fb2-dcb0-4fc4-adbf-fabbfc509456): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.893118 [ 31690 ] {} <Debug> shard_0.from_1 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.893200 [ 31690 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeRestartingThread): Execution took 281 ms.
2026.05.06 08:50:33.893294 [ 32120 ] {} <Trace> shard_0.from_1 (703d9b0c-8b0f-48d1-82ad-60dea3b6b5f7): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.893732 [ 31688 ] {} <Debug> test_1.insert_dedup_token1 (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.893807 [ 31688 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeRestartingThread): Execution took 281 ms.
2026.05.06 08:50:33.893849 [ 32085 ] {} <Test> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.893943 [ 32085 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:50:33.894035 [ 31712 ] {} <Debug> shard_0.to (ReplicatedMergeTreeRestartingThread): Table started successfully
2026.05.06 08:50:33.894051 [ 32085 ] {} <Trace> test_1.insert_dedup_token1 (0d2b666f-a2a0-48ee-95c0-9c76f867f0f1): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.894122 [ 32124 ] {} <Test> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf) (MergerMutator): Nothing to merge in partition all: Did not find any parts to merge (with usual merge selectors)
2026.05.06 08:50:33.894157 [ 31712 ] {} <Trace> shard_0.to (ReplicatedMergeTreeRestartingThread): Execution took 284 ms.
2026.05.06 08:50:33.894250 [ 32124 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf) (MergerMutator): Checked 1 partitions, found 0 partitions with parts that may be merged: [] (max_total_size_to_merge=161061273600, merge_with_ttl_allowed=true)
2026.05.06 08:50:33.894384 [ 32124 ] {} <Trace> shard_0.to (a55ce901-4bf5-4668-b420-14c5d0a694bf): Scheduling next merge selecting task after 60000ms, current attempt status: CannotSelect
2026.05.06 08:50:33.896332 [ 32135 ] {} <Trace> shard_1.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 15058ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:33.897219 [ 32101 ] {} <Trace> shard_0.from_0 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 17676ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:33.897417 [ 32102 ] {} <Trace> shard_0.from_1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 19535ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:33.897664 [ 32086 ] {} <Trace> test_1.insert_dedup_token1 (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 60000ms (points: 0, interval: 72292ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:33.897754 [ 32081 ] {} <Trace> shard_0.to (ReplicatedMergeTreeCleanupThread): Scheduling next cleanup after 1000ms (points: 0, interval: 17082ms, ratio: 0, points per minute: 0)
2026.05.06 08:50:34.164191 [ 31631 ] {} <Fatal> Application: Child process was terminated by signal 9 (KILL). If it is not done by 'forcestop' command or manually, the possible cause is OOM Killer (see 'dmesg' and look at the '/var/log/kern.log' for the details).